Enrich Dataset

After completing these steps, you will have created a new dataset using self-service data preparation. This new dataset will help to isolate invalid claims.

1. Click ‘More Actions’.

   

2. Select ‘Prepare Data’.

   

3. The self-service data preparation room shows up.

   

4. The first record of the data is actually the column header.

   

5. Check ‘Use first row as header’.

   

6. Click ‘Continue’.

   

7. The application will automatically recreate a new sample with the updated metadata structure.

   

8. The dataset now has the proper header.

   

9. Click ‘Actions’.

   

10. Click ‘Enrich Preparation’.

    

11. The enrich preparation user interface opens.

    

12. Click ‘+’ to add a new data source to merge with.

    

13. Click ‘Browse’.

    

14. Select ‘HANA_HC’ as a ‘Connection’.

    

15. Click ‘PHARMA’.

    

16. Select ‘PHARMA_CLAIMS’.

    

17. Click ‘OK’.

    

18. The application is acquiring a sample of the newly selected dataset.

    

19. The newly selected dataset can now be joined to the existing dataset.

    

20. Drag and drop ‘PHARMA_CLAIMS’ on the cell on the left hand-side of the main dataset.

    

21. Select ‘Left Join’.

    

22. Scroll down the list of output columns and uncheck ‘ORIG_PRODUCT’, ‘POTENCY’, ‘DOSAGE’, ‘ROUTE_ADMINISTERED’, ‘NOTES’.

    

23. Click Apply.

    

24. The application displays a preview of the merge data.

    

25. The merged data now shows a null value for the column ‘DRUG_NAME_0’ when a record from the claim data is for a drug that is not listed in the list of supported drugs.

    

26. Click ‘Apply Enrichment’.

    

27. The main self-service data preparation room now shows the enriched dataset.

    

The enriched dataset now contains null records for the field ‘DRUG_NAME_0’ for the records in the claim dataset which the drug name did not exist in our reference.

Potentially, there are multiple reasons for that. Some might be spelling mistakes of drug names, some other might be drugs that are not taken into account by the insurance company, some could be that the drug name in our claim was null.

You can now use this enriched dataset to isolate the data quality issues to further understand the data.

28. Click ‘Actions’.

    

29. Click ‘Add Columns’.

    

30. Type ‘ValidClaim’ for the ‘Column Name’.

    

31. Click ‘Expression’.

    

32. Type the following expression:

33. Click ‘OK’.

    

34. Click ‘Apply’.

    

35. A new column is now created.

    

36. Select the column ‘DRUG_NAME_0’.

    

37. Click ‘Remove’.

    

38. The column ‘DRUG_NAME_0’ has been deleted.

    

39. Click ‘<’ to navigate back to the ‘Actions’ menu.

    

40. Click ‘Run Preparation’.

    

41. Type ‘PHARMA_CLAIMS_ENRICHED_’.

    

42. Click ‘Apply’.

    

43. Click ‘Data Intelligence Metadata Explorer’.

    

44. Select ‘Monitor’ and click ‘Monitor Tasks’.

    

45. The ‘Monitoring’ application displays all tasks. Click on the ‘Preparation’ tab to filter on preparation tasks.

    

46. Filter on your task (i.e. DRUG_). Wait for your task to complete.

    

47. The task is completed.

    

48. Click ‘Data Intelligence Metadata Explorer’, and click ‘Home’.

    

49. Click ‘Browse Connections’.

    

50. Click ‘DI_DATA_LAKE’.

    

51. Navigate to the ‘shared’ directory. Click ‘Pharma’.

    

52. Type ‘’ in the Filter field.

    

53. Click .

    

54. Click ‘More Actions’ on the newly created dataset named PHARMA_CLAIMS_ENRICH_.

    

55. Select ‘View Fact Sheet’, Click ‘Overview’.

    

56. The factsheet for the dataset is not profiled and not published.

    

57. Click the ‘Profiling’ icon.

    

58. Click ‘Yes’.

    

59. Wait for the profiling to be executed (there will be two notifications which you can check by clicking on the notification icon). Then Click ‘Refresh’.

    

60. The dataset is now profiled.

    

61. Click ‘Data Intelligence Metadata Explorer’ and Click ‘Home’.

    

62. You returned to the Metadata Explorer home page.

    

---

Well done! You’ve now used Metadata Explorer to create a new enriched dataset using self-service data preparation. This new dataset helps to isolate invalid claims.

---