In a scheduled job, does Dataprep take a new file each run?

[46238cd6fba4c436815e]

When I schedule a dataprep job for a certain flow and I'm importing the dataset from a GCS path, each time the job runs would it take just the new file added to GCS and apply the recipe to it or take all of the files from that GCS folder? If the later is true, is there a way to schedule a job that would be executed only upon a new file added to the source folder?

I can't find clear explanation for this in the documentation, would appreciate a clarification!

Thank you!

[AMiller_Tri · Accepted solution]

Hi @Marija Stojkovska? 

When you schedule a dataprep job for a certain flow, it depends on the input you've given it.

1. If a specific file's been given as an input - that specific file would be taken, each time
2. A folder was chosen (through input-file parameterization), but no specific pattern was given (E.G - * [joker]) - All the files will be used as an input source - after being automatically unioned
3. You can, through input parameterization - have the job run only on files that abide a pattern. E.G - if files have a timestamp within their names - an input-path parameter could contain "yyyymmdd", describing the last day\week\month of your choice. This way, when the schedule is being run - only files with the names that match this timestamp will be used as an input datasource.

Alternatively - if you'd want a job to be executed only upon a new file is added to the source folder - I'd advise you to read Victor's Blog, regarding "How to Automate a Cloud Dataprep Pipeline When a File Arrives"

https://medium.com/google-cloud/how-to-automate-a-cloud-dataprep-pipeline-when-a-file-arrives-9b85f2...

After reading this article, you will be able to drag and drop a file in a folder, get your entire data pipeline executed and loaded in your data warehouse, and have up-to-date data in your reports and dashboards with a few simple clicks.

Hope this helps,

Amit.

[46238cd6fba4c436815e]

Thank you @Amit Miller?. I'm trying the last option you suggested.

[AMiller_Tri]

My pleasure.