If you source data from a system that you (warehouse team) have no authority to enforce data quality at the source level, how do you handle this? Do you apply data validation just after you stage the data? Or do you enforce it at the “end” of the flow (typically as you’re pushing it into the data mart)? Or do you not enforce validation at all?
RE: If you source data from a system that you (warehou...
One of the main purpose of Datawarehousing is to get Cleanse data. So skipping data validatons is not a good idea. It will be better to bring the data to a staging area and perform validations. Or while pushing data from the source we can perform data validations.
RE: If you source data from a system that you (warehouse team) have no authority to enforce data quality at the source level, how do you handle this? Do you apply data validation just after you stage the data? Or do you enforce it at the “end”
A DW is no longer relevant when it contains inconsistent data. The strength of a DW - and its robustness - also relies on the validation processes... The sooner you can detect and report any inconsistency the better it is. Wrong data must be trapped as soon as possible within the ETL processes.
