Submit your salary here

i work for a consulting company and i'm working in a client that has a lot of their data in old on-prem environments, being more specific, oracle, sql server, postgres.

their workflow is pretty basic: run ADF to collect data from these sources and send into ADLS, then use Databricks to process it. we're planning to modernize their environment (using unity catalog and other new stuff) and one of the things we're thinking is to retire the ADF

the reason is simple: ADF is a pain in the ass (we're having a hard time working with it specially because it was another company that build all that shit, the connection with Azure DevOps/Repos is always horrible to manage and the workflow itself needs an upgrade); and because Microsoft is pushing hard the Fabric Data Factory

i know that Databricks with Lakeflow Conn can connect to on-prem using express route, VPN, but in scenarios where this could not be possible, what tool could be used to send data from on prem to azure data lake?

i like to code so my first suggestion was to use local airflow and simply read from db and upload to adls. it's free, it's versionable and has tons of documentation; but i'd like to test other options before suggesting anything

i was reading about airbyte, the pros and cons of the tool, and looks interesting

how do you guys handle this kind of workload?

Full disclosure: I work at Astronomer and I'm one of the judges.

Beyond the Dag is a data engineering hackathon with an Apache Airflow focus. Take note: this is purely about open-source Airflow!

It's all about showing modern Airflow has evolved beyond just moving data.

Treat Airflow as a platform, extend it with plugins, add agents with the new Common AI provider, and keep a human in the loop where it matters. Use the tracks as inspiration, not limits, and aim for something that shows what Airflow is really capable of.

Why am I sharing this with the community?

I love exploring creative ways to learn frameworks and technologies. In the past, I've built some unusual projects, including gesture-controlled Airflow, native retro game emulation within Airflow featuring Doom, Age of Empires, and others, as well as a Minecraft PaperMC plugin that lets you build Airflow Dags inside Minecraft. I've also participated in many hackathons, and my experience has always been that I learned a lot and met some great people.

Another reason is that hackathons with a clear DE focus are rare, making them valuable additions to your profile. For example, participating in those helped me during my U.S. visa application process.

Because of that, I think this is a great project for learning about Airflow, having some fun, strengthening your data engineering profile, and maybe even winning a Switch 2 😉.

The build window already started and is open until September 24. We kept it relatively relaxed, so it's still time to join. There are also prizes, including a Nintendo Switch 2.

More details and registration at:

https://www.astronomer.io/events/beyond-the-dag-data-engineering-hackathon-2026

I recently started my first ever full time role in data engineering. However, it appears that the big data stack for my company has already been established.

Therefore I’m dealing more with bug fixes and new feature implementations but the stack isn’t likely to change unless I make a good case for it. Furthermore, my company heavily depends on AI like Claude and Codex to assist with solving these tasks, which offloads a lot of the coding to these agents.

What kind of recommendation would you senior devs give to a fresh grad like myself? I feel my coding competency going down the more I work. However, since these tasks are offloaded to AI, I do believe that my understanding of data flow in big data architecture has deepened quicker.

Our stack is supports both streaming and batch processing if it matters.

Hi community.

At work, we’re facing a bit of a challenge: we receive a huge volume of JSON-format events from multiple microservices into Kafka topics. We want to build a dashboard to monitor these topics and alert the business if we don't receive the expected events (we’ve already identified specific projects and know the expected TPS for each). Do you know of any monitoring tools that might work for us?

Thanks in advance!

Those who went fully remote, what's your thoughts

I currently work hybrid 2 days a week in the office in London which is fine, I'm principal data engineer and this gives me some face time with direct reports and the regional ones I influence. But it's at least 6 hours a week commute and £6k train tickets a year.

I'm looking at a fully remote role, I wonder whether I'll feel the downsides of that. Are you forgettable as a fully remote? Do you miss office time? It's it hard to have influence?

What's your experience?

I have been thinking of studying for the CDMP exam- I am a data analyst but want to transition data governance and in the future to AI governance. Would the CDMP or some other certifications help?

I’m currently a DE for a major retailer (think Kroger, Walmart, Albertsons, etc.), working on the pricing and product data streaming side. The WLB is amazing, and I’m earning around market rate for a junior DE in a major city. However, I’m starting to gain interest in high-frequency data streaming for large brokerages and financial firms.

For those who have made a similar move, how did you manage to switch industries, and which industry do you believe is the most lucrative for a data engineer?

I’ve been in tech for over 20 years and I have never in my life used python, or any other programming language, to manipulate data.

I’ve manipulated data in every single way you can imagine, including parsing out semi structured files, such as json and xml using SQL alone.

python should be used to handle FILES themselves or make API calls. things of that nature.

why is this field such a horrendous mess? not to mention, Python is a horribly performant language. if you care about performance you should be using C#

I'm a data engineer at an e-commerce org, and I'm fairly happy - we have interesting problems, large volumes of data, and plenty of initiatives on our backlog. I largely get to work on what I want, at my own pace, and choose how things are implemented.

But I am getting tired of doing everything with the end goal being to sell more things and increase profit extraction from consumers.

Who is working in interesting data related fields/careers that provide essential goods or services to the public/planet/humanity. Keen to hear any stories or opinions!

EDIT: thanks for all the interesting responses!

My company wants to replace Fivetran with easier and cheaper alternative.

Context & Stack:

• Stack: Snowflake, dbt, Fivetran
• Major Ingestion: Salesforce (4 production instances) -> Snowflake

We are looking for alternatives to Fivetran due to escalating ingestion costs. We strictly need an ingestion tool (we handle all transformations via dbt).

What We've Tried (and why it failed/didn't fit):

• Matillion: Failed at our scale (couldn't handle 15-minute syncs).
• Airbyte: Failed at scale.
• ETLWorks: Cost.
• Informatica / IDMC: Already in-house, but trying to sunset due to high costs.
• Airflow: Already in-house, but too much custom configuration overhead.
• Boomi: Already in-house, but on-prem and costly.
• Hevo Data: Lacks the necessary scale.

Our Current Top Contender: Openflow is being heavily pushed by upper management. They have a soft spot for it, likely because Snowflake is making it lucrative/incentivizing adoption to build their user base.

Our Core Requirements:

• Ingestion-only focus (dbt handles transformations) (imp)
• High scalability and cloud-native performance (imp)
• Fast execution with simple configuration, yet deep integration options
• Rich catalog of connectors
• MCP and AI capabilities (optional)

Do we have any other realistic solutions or shall go with Openflow?

We are okay to try modern solutions like utilizing skills / agents to create connectors on the fly etc but these solutions we have known to be not much scalable..

Any help is appreciated..

Im a data engineer with 4 years of experience, and In getting pretty bored with my current job. Most of the code is generated by AI now, so my dayis mostly just maintaining pipelines.

I ve been offered an internal move to Data Product Management. Its for one of the biggest projects in the company and will be built on Databricks. They told me they need someone with a solid data engineering background who can translate business requirements effectively.

However, I have a few doubts
Am I just going to end up as a ticket creator? (Im not entirely clear on what a Data Product Owner actually does)
Is this really a step forward in my career?
I know for sure I will get more exposure to senior leadership, but Im still on the fence.

Hi guys, I'm a junior AI and DS person with a masters in both, job searching, and had been finding myself a bit mismatched for the nature of the work I was previously in - constant stakeholder management and convincing, non technical meetings, presentations, not to mention, just being overwhelmed with so many new AI technologies and tools, etc.. I prefer structured and calm working environments and figured that, after getting some interviews for DE roles and reading about the job responsibilities/researching a bunch, DE might be better suited for me. I understand that staying up to date with new tools will apply regardless of the field, but it seems a lot calmer? I think it's also a great foundation to build on.

I wanted to know the day to day of what a DE actually looks like, how much of it is stakeholdery, requirements gathering, coding, etc.? Let me know what field you work in or how this varies by industry, and maybe also about your career progressions.

Wondering if I have an advantage here and should just do the jump or if staying in AI/DS is the better move rn.

Thanks!

hi there. i work as data engineer. previously worked as analytics engineer / analyst. i no longer enjoy spending whole day writing code and looking for a switch to a more client facing role where my technical knowledge will be beneficial. i’m not bad at communication. anyone been in similar position?

built this free sandbox because cloud costs were killing my learning, sharing in case it helps someone else

https://github.com/haltinok/oss-da-bi-stack

G’day!

I have a few Snowflake Tasks that run daily and I was finding it tiresome/cumbersome to jump into Snowflake to track down what went wrong/double check details… so I built a tool to do it quickly from the command-line:

https://github.com/ben-nour/snowtask

It’s fairly basic but you can filter rows using regex, tailor the tables to display the columns that you want to appear (one of my paint points with SHOW TASKS is that you can’t properly specify only displaying the fields you want) and order the columns.

I hope it’s of use to someone out there!

Note that I do NOT work for Snowflake.

I am building a social, and here are the requirements:
-A user can create posts and a post can have multiple media (photo/video)
-A user can create a project and a project can have multiple media
-A project can become a post
-A user will have their personalized feed

the initial design I planned is:
tables:
user | post | project | media|

relations:
user one->many post

user one->many project

post2media junction table

project2media junction table

the idea here is to reuse the media uploaded in projects so that when they are shared as posts, we reuse the same media links rather than uploading new duplicate media.

The problem is that these junction tables will slowdown the feed generation especially

I am using postgreSQL, and I am thinking i should not using it to generate feed, maybe use a NoSQL DB/ Graph.

The question is how should I design my postgreSQL DB, so that feed generation does not break my app?

Do I need NoSQL DB and how would I use it in my case?

How should I store media ?

We created this quick tool for visualizing and modifying parquet files, enjoy! Please let me know if you have any feedback or questions :) (this is the company i work for, they pay me US dollars in exchange for labor)

Looking to understand what your org structures look like? Is it a small company or a large company? What teams do you have? How do you separate responsibilities? What does support and deployment look like?

I have a 12 YOE with a focus on DevOps, some python, AWS Cloudformation and MongoDB. Lately I've been less hands on but have worked on event driven architecture design and done code reviews for pipelines running with serverless components and Python SDK. Most of the data I've worked with has been csv and spreadsheet data for schedules, structured metadata and image/media key value stores. As a result I've primarily worked on MongoDB and used aggregation pipelines to join or transform data for downstream deployments. I have also used Gemini Pro and couple of POC deployments of Ollama for a RAG application (non prod).

I am now looking to get a crash course on Data engineering and databricks, but a little confused whats the best way to get a good understanding of typical data engineering problems (I have a vertical I need to focus on so looking for data patterns that I can then translate to what I need), what gaps I need to fill having no experience with Databricks, little to SQL and any data warehouse technologies. I've not used dataflow or kinesis etc yet so I dont have hands-on experience with these type of streaming pipelines either. (Claude has given me some good insights but reddit often times has better more real world recommendations)

Are there Udemy courses or any other video series that is considered gold standard for onboarding? Would also be open to some blog posts or project ideas to get my feet wet. ideally if AI or ML based applications would be ideal. Cheers!

Hello, community! I’m an ETL developer specializing in Ab Initio, and for the last 10 years, I’ve been working for a bank in Mexico.

I thought my position was “secure” because of all the legacy systems they had in place. Unfortunately, that wasn’t the case. When I started looking for other positions, I realized that I wasn’t familiar with many of the technologies employers were asking for.

If anyone out there has faced a similar problem or concern, maybe this book I wrote could be useful to you.

You can get the book directly from Amazon:
https://www.amazon.com/dp/B0HG6XKG2S

Thanks for taking the time to read my first post! 😄