Use Dataprep through API and Python language

[43e0a22239487a85233b]

Is that possible to interact with Dataprep through API or Python so I can execute transformations through my programs, instead need to login to web portal and do all the wrangling in web interface ?

[Trifacta_Alumni]

Hi Gustavo,

Dataprep have REST API you can use to start Dataprep jobs from an external script or tool.

API doc https://cloud.google.com/dataprep/docs/html/API-Reference_145281441

Examples with Cloud Functions https://github.com/victorcouste/google-cloudfunctions-dataprep

Example of data pipeline using Cloud Function and Dataprep API https://medium.com/google-cloud/how-to-automate-a-cloud-dataprep-pipeline-when-a-file-arrives-9b85f2...

Victor

[43e0a22239487a85233b]

thanks a lot @Victor Coustenoble? really appreciated

[43e0a22239487a85233b]

Hello @Victor Coustenoble? your tutorial at medium about a pipeline is great, im trying to create my pipeline and when im trying to parameterize my FileName variable to take all *.csv files in a storage it is not allowing me to update the parameter.

See the picture. Am I doing something wrong?

[43e0a22239487a85233b]

@Victor Coustenoble? another issue is regarding the code, i did the necessary adjustments (token, landingzone, etc) but is getting this error:

"Result : {'code': 16, 'message': 'JWT validation failed: Bad JWT format: should have 2 dots', 'details': [{'@type': 'type.googleapis.com/google.rpc.DebugInfo', 'stackEntries': [], 'detail': 'auth'}]}"

[Trifacta_Alumni · Accepted solution]

Hi Gustavo,

For your first question, you need to use Pattern parameter (with wildcard for example for your need). Note that you can mix date, pattern or variable parameter type :

For your second question, difficult to say, there must be an error in your source code.

Happy to have a look if you send it to me at vcoustenoble@trifacta.com

Victor

[43e0a22239487a85233b]

Ok I got, but if I will use wildcard is it going to impact my code since it will not have FileName variable anymore, right?

[Trifacta_Alumni]

Hum, yes in this case you don't need anymore a variable, and I think your code won't work as you will try to overwrite a variable that won't exist anymore.

Don't know how you could easily switch between a Pattern parameter and a Variable parameter, as it's not manage in the same way.