Data Remediation

After completing these steps, you will have fixed data quality issues using the data remediation preparation.

1. Click ‘View Preparations’.

   

2. Click ‘Preparation for PHARMA_CLAIMS_ENRICHED_.csv’

   

3. Click the header for the column named ‘DRUG_NAME’.

   

4. Click ‘Replace’.

   

5. Type ‘Nystatine’ for the search string and type ‘Nystatin’ for the ‘Replace by’. Click ‘Apply’.

   

6. The value ‘Nystatine’ was replaced by ‘Nystatin’. Click ‘Replace’.

   

7. Type ‘Acetaminofen’ for the search string and type ‘Acetaminophen’ for the ‘Replace by’. Click ‘Apply’.

   

8. The value ‘Acetaminofen’ was replaced by ‘Acetaminophen’. Click ‘<’ to go back to the ‘Rulebook Dataset Results’.

   

9. Click ‘<’ to go back to the ‘Run Preparation’.

   

10. Type ‘REMEDIATED_DATA_’ for the ‘Dataset Name’. Click ‘Apply’.

    

11. Click ‘Data Intelligence Metadata Explorer’. Click ‘Home’.

    

You fixed data quality issues using the data preparation recipe that was automatically generated with the execution of the rulebook. In a normal process, the output dataset resulting from that data remediation process would be used in order to update the original table that contains the data quality issues using workflow and approvals. For this hands-on, you can use the data preparation recipe you created to simulate the update process of the data and see the results in the rulebook.

12. Click ‘View Preparations’.

    

13. Click ‘DRUG_’.

    

14. Click ‘Recipe’. Edit the ‘Enrich’ action.

    

15. Click ‘+’ to add a new data source.

    

16. Click ‘Browse’. Select the connection ‘DI_DATA_LAKE’.

    

17. Click on the ‘shared’ -> ‘Pharma’ folders.

    

18. Search for the ‘’ folder. Click your folder.

    

19. Click ‘REMEDIATED_DATA_’.

    

20. Click ‘OK’.

    

21. The remediated dataset was added to the list of sources.

    

22. Click ‘X’ to delete the join with the dataset ‘D1’.

    

23. Drag and drop the ‘REMEDIATED_DATA_’ data source to the left side of ‘P1’ .

    

24. Select ‘Left Join’.

    

25. Scroll down and uncheck ‘VALID_DRUG’ and ‘POTENCY’ under the ‘left source’ list of columns.

    

26. Scroll down and uncheck ‘ORIG_PRODUCT’, ‘DOSAGE’, ‘ROUTE_ADMINISTERED’, and ‘NOTES’ from the right source list of columns. Click ‘Apply’.

    

27. Click ‘Apply Enrichment’.

    

28. Click ‘POTENCY_0’. Click ‘Rename’.

    

29. Type ‘POTENCY’ for the ‘New Column Name’. Click ‘Apply’.

    

30. Click ‘<’ to come back to the ‘Run Preparation’ menu.

    

31. You can now run your preparation to create a new curated dataset:

    • Type ‘PHARMA_CLAIMS_CURATED_’ for the ‘Dataset Name’.
    • Click ‘Apply’.
      

32. Click ‘Data Intelligence Metadata Explorer’. Click ‘Home’.

    

33. Click ‘rulebooks’.

    

34. Type ‘PHARMA_CLAIMS_’ in the ‘Filter rulebook names’ text field. Click on your rulebook.

    

35. Expand both ‘Accuracy’ and ‘Completeness’ set of rules.

    

36. Edit the rules binding associated to the first rule: ‘DRUG_NAME_VALID_’.

    

37. Delete the rule binding and click ‘Save’.

    

38. Click ‘Close’.

    

39. Expand ‘Accuracy’. Click ‘Create Rule Binding’.

    

40. Click ‘Step 2’.

    

41. Click ‘Browse’. Select ‘DI_DATA_LAKE’ for the ‘Connection’.

    

42. Click ‘shared’. Click ‘Pharma’. If you get a ‘Container is not supported by rules’ message then click directly on the folder name.

    

43. Filter ‘’ and select your folder. If you get a ‘Container is not supported by rules’ message then click directly on the your folder name.

    

44. Select ‘PHARMA_CLAIMS_CURATED_’. Then click ‘Step 3’.

    

45. Select ‘VALID_DRUG’ for the mapping. Then click ‘Create Binding’.

    

46. Click ‘Close’.

    

47. Expand ‘Completeness’, then edit the rule binding.

    

48. Delete the column mapping for ‘DRUG_NAME’. Then click ‘Save’.

    

49. Click ‘Close’.

    

50. Expand ‘Completeness’, then click ‘Create Rule Binding’.

    

51. Click ‘Step 2’.

    

52. Select the existing ‘Connection ID’ for ‘PHARMA_CLAIMS_CURATED_’, then click ‘Step 3’.

    

53. Click ‘Create Binding’.

    

54. Click ‘Close’.

    

55. Click ‘Run All’.

    

56. Click ‘Run’.

    

57. Wait for the rulebook execution to finish.

    

58. Click ‘View Results’.

    

59. Expand the dataset.

    

60. The quality of the data improved.

    

---

Well done! You have now fixed data quality issues using the data remediation preparation.

---