I'd love to be able to drop empty columns.

[4a0098b7f9b243add27a]

Or even a rule based column drop that extends beyond just empty. But empty would eb a great start.

[Trifacta_Alumni]

Hello Jonathon, could you provide more information for how you would ideally like to see this function being used and your use case? This is so our internal team can have more context and understanding of the requirement. Thanks.

[Trifacta_Alumni]

I would hazard a guess that the context is based on the following:

https://community.trifacta.com/s/question/0D51L00005YoywgSAB/is-there-a-column-limit-on-csv-ingestion?t=1543589427492

I will file an enhancement request on your behalf.

[4a0098b7f9b243add27a]

Apologies @Gina Weng? i missed your reply.

@Nathanael Kuipers? that is part of it yes, i have some very sparse datasets that are 'knowable' in terms of the formulation of the column names but the number of columns are 'unknowable'. Many of these columns may also be empty.

I'm hoping for a 'least- manual' workflow to amend these or make repeatable. Among the features that would make sense to me:

1. Drop empty columns
2. Regex rename Columns based on text pattern
3. Select Columns based on text pattern for filtering or transformation.

The latter two are related to the issue you referred to. Building recipes in the UI is at time painful when wishing to unpivot, drop, filter replace and other actions possible over multiple columns in this situation.

e.g.

• csv has 50 columns headed "Label"
• dataprep appends these incrementally Label, Label1, … Label49
• recipe step may remove rows that contain only data in any of those 50 columns
• attempting to e.g. unpivot these 50 columns requires selecting each column individually in the UI
• if there are 49 this time and 51 next time, any schedule will not run

If the column selection could instead be via /Label\d{0,}/ say then this would consistently be successful.

The part request of dropping empty columns relates more to reducing the number of columns from which to manually select via UI.

Thanks

[Trifacta_Alumni · Accepted solution]

Hi Jonathon,

Thank you for the additional information. Have passed it along, and the feature request has been acknowledged by engineering.

Cheers,

Nathanael

[4a0098b7f9b243add27a]

Thanks. It’s a nice to have. In some instances I have flows that daisy chain pivots unpivots and joins which has helped a little.