top of page
Search
  • Writer's pictureTim Burns

Machine Learning will change everything

Updated: May 16, 2023


Our world is filled with layers of information - waiting to be comprehended (Author)


I have been researching using Machine Learning to build domain models, quality controls, and natural language analysis around data pipelines. The deeper I delve into ML, the more convinced I am that it will radically change how we build data pipelines, even more than the recent transition from on-prem solutions to cloud-based solutions.


Business data needs to grow faster than many data engineering teams can keep up. Data engineers have many tools: Snowflake, AWS, Terraform, and dbt. However, orchestrating meaning and action in the data pipeline remains a persistent problem. Natural language processing engines like OpenAI offer an automated mechanism to connect components logically without human intervention. As a result, analysts and engineers can supervise the process of turning data into value, make connections extending beyond siloed domains, and ultimately build better data products.


Quote from C. Samiulla's reference on testing and monitoring.

In this way testing & monitoring are like battle armor. Too little and you are vulnerable. Too much, and you can barely move.


References

  1. D. Le et al (2020) Baselines

  2. B. Mathes et al (2021) ML Metadata: Version Control for ML

17 views0 comments

Recent Posts

See All

Carto, Snowflake, and Data Management

A basic principle of data management: Don't move data unless you have to. Moving data is expensive and error-prone. Data Egress Cost: How To Take Back Control And Reduce Egress Charges Archiving to S

Music Festivals 2024

I am browsing music festivals I'd like to attend in 2024. I work for Evolve.com which manages vacation properties and my search came up with this site from one of our biggest competitors. Vacasa: US

bottom of page