Photo by Author of the Never Summer Mountain Range
Many years ago, I had a colleague who had a similar role I now have: Build and maintain a Data Warehouse. Like me, his title didn't match his job, he was titled Database Administrator, and I am titled "Software Engineer." Like me, he was tasked to build a Data Warehouse using the Kimball Model, but in Oracle and not Snowflake.
My role was as a Web Developer, and it was far more prestigious than DBA. At that time, a web presence was new, and tools for building analytics were primitive and expensive. I built the web pages and our reporting interfaces to a database. I used Java, Tomcat, and Struts. I remember being so proud of myself for learning struts and implementing a "Model View Controller" pattern, thinking I had arrived as a software engineer. In retrospect, all the work I did studying Object Orientation, Design Patterns, and everything that focused on the software system (refactoring, test-first development, polymorphism, etc.) was a complete waste of time.
At that time, more so even than today, database developers were undervalued. Back then, there wasn't even the title "Data Engineer." The database developer was a DBA or even a PL/SQL developer but not a true software engineer. The PL/SQL developer was a far lesser breed than the "Object Oriented" Java or C++ developer. They were paid less and didn't use the impressive language found in books on "Design Patterns: Elements of Reuseable Object-Oriented Software" to describe the systems they built.
Rolling ahead twenty years, Data Engineer is a respected title and not the half-educated journeyman for "computer scientists" to perform the lesser "database work" of a software product. The greatness of real data engineers like Werner Vogels and the realization that all problems are data problems have changed the value proposition of the data engineer.
As for Design Patterns and Reusable Object-Oriented Software? Nobody gives a shit about that anymore. It was a fad. Now we build out architectures using templates, and the security holes of a Python project can be gauged by the amount of worthless 3rd party libraries that are stacked in using those same design patterns.
Back to my colleague, I remember how he cycled through hire after hire for over four years. Hiring them and firing them. It wasn't until a fellow senior engineer, and I took the new colleague out to lunch to get to know them and said to us, "I can program circles around you." Yes. I know he had found his developer. That level of arrogance to tell the two most senior developers in a company that they sucked in comparison was an indicator that he had found his developer. She proved it over time, and by getting every certification, tackling every data problem, and delivering the system's core features, she showed she was one of those developers worth keeping. And, sitting across the table from her, we were full of shit with our "Object Orientation" and Martin Fowler nonsense. In retrospect, it was the Data Engineering (PL/SQL and Oracle) she and her boss did that built the Oracle data system that founded the company's growth, which finally became IRI, a $1.2 Billion dollar company.
Fast forward twenty years, Data Engineer is a real title. Every startup with ambitions of billion-dollar valuations positions itself as a data company. Alas, good engineers are still as difficult to find. Despite the nonsense of the recruiting industry, 9 out of 10 "Great Senior X Engineer" candidates lack basic development skills like knowing how to program their way out of a wet paper bag. Of the 1 in 10 that do have the skills, only 1 in 10 has the dedication and humility to take the necessary steps to be an excellent Senior X Engineer. Programming a language, be it SQL, Python, Java, .NET, etc, is just a start. The real value of a software engineer comes from humility in terms of devotion to the business. Not the humility of not being arrogant. If you are good, as the Oracle Data Engineering team was from my earlier job (mind you, I wasn't on this team: I was one of the idiots), you are right to be arrogant. Especially if the software engineers and fiddling around with 0-value activities like refactoring, mock objects, unit tests, etc.
Real Data and Software Engineering starts with being able to code anything in your chosen language. Beyond that, you need project management skills. Yes, you are the project manager of your project. You need analyst skills. You need to pick up an Excel spreadsheet and detail exactly what you need. And finally, and most importantly, you need to be responsive to your customer. Throwing your code over the fence and considering your time "too important" to deal with directly will stunt the value of whatever you produce because it will be about some computer science bullshit rather than addressing a real-world problem.