The Evolution of The Data Engineer: A Look at The Past, Present & Future | Airbyte
Hello Hacker News ! We're so happy you liked our post. We're currently running the State of Data Engineering survey ! Whether you work in data, have opinions about Airflow vs Dagster, or want to shout out your favorite Data community, substack, or YouTube creator, or just think Data has become overcomplicated, come fill out our survey !
The art of drawing lines
With many data teams managing hundreds or thousands of data models, the complexity of today's data stack keeps exploding. To make it all more complicated, data often freely flows through various systems lacking explicit definitions, structure, and guarantees. Yep, I am going there-Data Contracts. Enough has been said about what the data contract is.
DX Is the New UX
In the past decade, most of the innovations in the data ecosystem were focused on producing more data. It worked, and just as UX became important as digital products proliferated, DX has taken an important place as data products have burgeoned. In fact, 20 years ago, data was the realm of IT teams and a few data wizards.