Is there any limit in the number of source data records (undergo transfromation) that can be downloaded from trifacta UI once the job is completed?

[41698a0ca29453be25d2]

For example, if there are 1 Million records in source and on running the job, there are 1 million records . Can we download the transformed file from trifacta's jobs page with all the source records?

[Trifacta_Alumni]

Hi @SRUTHY S?,

If one of the job outputs is a file on the base storage layer (e.g., CSV), then you should be able to download it from the Jobs page (i.e., click Jobs tab on the left, Job ID, Output Destinations tab along the top, hover over the output and click the context menu => Download results). I do not believe that this varies with the size of the output.

Why do you ask?

Regards,

Nathanael

[41698a0ca29453be25d2]

Thank you for your response.

So if I understand correctly, there is no limitation in the number of source records that can be processed and the number of records that can be downloaded from trifacta UI.

[Trifacta_Alumni · Accepted solution]

Well, there is a world of difference between processing (i.e., ingesting +/- transforming + publishing) records, and downloading tabular results.

If you have a billion rows, then you certainly would not want to process them using Trifacta Server, for example. Rather, you would want an execution environment such as Spark. In theory, this can scale out in the cluster as needed, though there are a host of other properties that might need to be tuned in order to accommodate a given job. In practice, the answer to this question is therefore the limits of the available hardware.

But for downloading, the only limit, in theory, is how good your Internet connection is and how much time you have. A file is a file, no matter how large.

Does this help?

Nathanael

[41698a0ca29453be25d2]

Yes,thank you.