Transitioning Industries as a Data Engineer

[u/KirbyIsAName]

I’m currently a DE for a major retailer (think Kroger, Walmart, Albertsons, etc.), working on the pricing and product data streaming side. The WLB is amazing, and I’m earning around market rate for a junior DE in a major city. However, I’m starting to gain interest in high-frequency data streaming for large brokerages and financial firms.

For those who have made a similar move, how did you manage to switch industries, and which industry do you believe is the most lucrative for a data engineer?

[u/Astherol]

There are multiple ways - but you always need to remember how to negotiate your rates. I assume you may be like 3 years exp, 27 years old. Still having fun in technical stuff and caveats of streaming - go for it and don't look back, it's a long run, you will be doing best at things you feel obsessed about :) Having fun at analytics-DE border? Consider becoming Data Analitycs Engineer Looking for money and rate increase fast? Finance/fintech There are like 10 ways in front of you, if you are before 30 try to be comfortable and take pleasure at exploring it. My way to have fun is helping NGOs to not feel rapid burnout (I'm 33 with 10 y exp in Europe)

[u/Astherol → u/Astherol]

Tldr: Go for hobby project, have fun and make a commercial move afterwards ;)

[u/Key_Ingenuity5340 → u/Astherol]

Hey Astherol, do you mind expanding on the finance/fin tech aspect

[u/KirbyIsAName → u/Astherol]

Appreciate your response. I'm actually halfway through my first year as a DE as a new grad. Beside my main work, my side project involves incorporating additional attributes that us DEs source for the MLE team for product recommendations. How much Al/ML work have you been involved in during your tenure?

[u/Big_Arrival_626]

I don't think it will be that hard for you to switch! You have good experience plus work for a known company, now it's time to apply and network.

As for how lucrative: in general, I don't think the pay band is that different. You might end up making 10-15% more at a big bank. At higher levels, jpmc and Goldman have big bonuses which could make your comp more than 20-30% a major retailer. Big tech companies are more lucrative such as meta and Amazon.

Alternatively, you could work for Walmart since u already have retail industry exp and they honestly pay pretty well.

[u/lzwzli]

You have to think long and hard about how much you value WLB. That is one of the first things to usually suffer when you go to financial firms since every millisecond counts.

[u/andrew2018022 → u/lzwzli]

Data engineers have a lot more chill life at those companies. It’s not so bad.

[u/Unknown-333]

Your streaming experience should carry over nicely finance is similar just tick data and much tighter latency. Might be worth picking up the finance specific stuff (kdb+/q, low-latency Kafka/Flink). Finance/HFT tends to pay the most, but usually costs you the wlb you have now. Honestly sounds like youre in a great spot already chasing the comp is valid just going in eyes open on the tradeoff Good luck either way

[u/Shadowlance23]

I've worked in agriculture, energy, finance, and mining. Data is data is data. The core of the work is pretty much the same across the board so you won't have trouble with that.

You'll pick up the points unique to each industry pretty quickly.