Learning newly released tools as a developer can be exciting as it’s thrilling to be an early adopter, but it can also be frustrating. Often with new tools there is limited documentation and community support available which can cause you to doubt yourself. Instead you have to rely on yourself and your experience to figure it out. Like any new piece of software, bugs can pop up which make you question whether it’s your code or the tool causing some quirky behavior.
I got my start in the ETL world with Oracle Data Integrator or ODI. My next project used Informatica so I picked that up as well. I changed jobs and learned Red Point’s Data Management tool. I recently started getting my hands dirty using Azure, SSIS and Azure Data Factory. I like to think of myself as pretty well versed in ETL tools. While all tools are different and have their own nuances and tricks the underlying principles are the same. For the past couple weeks, I’ve run into some interesting features of Azure Data Factory. When something failed to work, my first thought was always that something was wrong with my code. This was a thought I would immediately have despite having over 5 years of data warehousing experience and having learned from and worked with some incredibly knowledgeable colleagues and experts. But here I am thinking, it’s me, there’s something wrong with my code.
I, like many others, have long suffered from impostor syndrome. For you non-psychology buffs out there, impostor syndrome is the feeling that sooner or later someone is going to figure out that I actually am not an ETL developer, that I have no idea what I’m doing or that I am a fraud. After working through these undocumented features, I started to realize that ‘it’s not me, it’s the tool.’ And maybe I’m not really a fraud.
The first notable bug I ran into was after creating external tables in my database, I was unable to then create normal tables with the same name. The tables I was actually going to use had to be named differently. I renamed the tables and then moved on. Next I went to use my external tables in a stored procedure, which also failed. For some reason, Azure Data Factory treats external tables as views which are unable to be referenced in a procedure. After going line by line through code, multiple times, I asked another team member if he had seen such an error before. He told me it was actually a bug. The trick to getting around this bug is to create your external table as a temporal table in your procedure. A few tweaks later, problem solved.
Using tools and products as they are being built is an awesome way to learn but can challenge your confidence as a developer. Sometimes you have to trick the tool to get it to do what you want and those tricks are sometimes less than obvious. Learning the tricks of the trade (pun intended) have helped me grow from feeling like an impostor to feeling like a developer. If the tools I’m using are used by developers everywhere despite having many bugs and being constantly under development then I can trust in my code and myself to know when it’s me and when it’s the tool.