r/dataengineering • u/DiscountStunning919 • 2d ago
Discussion why do people use software engineering languages to handle data when it is not the right tool for the job?
I’ve been in tech for over 20 years and I have never in my life used python, or any other programming language, to manipulate data.
I’ve manipulated data in every single way you can imagine, including parsing out semi structured files, such as json and xml using SQL alone.
python should be used to handle FILES themselves or make API calls. things of that nature.
why is this field such a horrendous mess? not to mention, Python is a horribly performant language. if you care about performance you should be using C#
13
7
u/Seerfer 1d ago edited 1d ago
Python is usually an interface for more advanced data processing tools (like Apache Spark) so language performance is not a problem. People use because it's easier for more complicated transformations when you need to organize it in functions, write loops or conditional statements
7
u/CorpusculantCortex 1d ago
Python is hardly a swe language. It is a well rounded toolbox. And it is commonly used in analysis and ds applications. It is a perfectly adequate tool for a lot of applications...
3
2
u/Outrageous_Let5743 1d ago
Well python is just a mere orchestrator. Every good data transformation tool is written in high peformance languages. DuckDB in C++, Polars in Rust, databases in C, Spark in scala.
1
1
u/Comprehensive-Tea-69 1d ago
Thank you for asking this question. I have thought about it like people who are not granted access or given the right tools just trying to get something done with what they have. Like how many analysts have absolutely ABUSED VBA in excel to do things that would be better handled at the database level.
But I’ll be interested what people say who actually do what you describe
2
u/disposepriority 1d ago
I’ve been in tech for over 20 years
if you care about performance you should be using C#
Don't let me catch y'all complaining about the job market again.
2
1
u/KobeBean 1d ago
Python is not an enterprise language, and I’d die on that hill. It works really well for solo or small team endeavors, so data science loves it.
1
u/Monad1 1d ago
To me it sounds like you have a terrific data engineering team, always delivering data in a queryable store. Fetching, transforming and orchestrating is a huge deal.
About python it's all about the ecosystem. Take a look at polars, arrow, spark for performance.
It always becomes a mess if the tools / methods are being misused.
1
u/Nekobul 1d ago
When the goal is speed of delivery (not execution speed), Python works well. However, that speed of delivery is later on paid in spades when you run your inefficient processes on the hyperscaler infrastructure. That means the hosting vendors themselves are also a likely culprit of the Python popularity. It is profitable to run the slop.
1
u/j0holo 1d ago
Parsing XML with SQL sounds like hell.
Programming languages often have libraries that make it more readable. And if you do any serious data parsing with Python you often use a library that does call C code.
If you really care about performance use a compiled language and understand your data, operations and hardware.
1
u/Outside-Storage-1523 1d ago
On spark, Python and SparkSQL are optimized enough that we don’t need to use Scala. Python is the de facto DE language IMO so it’s really easy to hire people who know it.
But I also agree that SQL is very good at writing queries, and as long as we are operating databases it doesn’t make sense to use pandas to read and transform data — I mean you can do it, but SQL is way more readable and faster.
1
u/dodovt Senior Data Engineer 22h ago
Then you realize most performant python libraries are written in C/Rust and you just do calls to them, meaning Python itself is mostly an orchestrating language with a good-enough abstraction that enough not-so-tech people can understand.
Hell, EVE Online is written in Python and all of its db management is python-heavy.
If they can get an MMORPG where over 5k people can be in the same node and parallel process all of the data between the interactions, how can you say python is bad and has terrible performance lol
1
1
u/Illustrious-Big-651 1d ago
I will also never understand why Python got a „data language“, it‘s crazy slow, type unsafe and memory hungry.
It‘s a decent scripting language but anything that should have high throughput suffers from its bad memory management and the GIL.
But I guess the reason is that all these ML libraries got offered as Python packages so people started doing their pre processing there.
And it‘s very often very bad quality code. The ML guys in our company don‘t care at all for memory usage and never heard of batching. They just request 120GB of RAM for a shiny app and think it‘s fine.
2
u/Outrageous_Let5743 1d ago
Data jobs did not just evolve from software engineers, but mostly from professors at universities. A good lot of them wrote python or R software to do data science. They can code but not on the same level a software engineer can. Now this also happens with data/maths students who take a couple of introductory classes of software engineering but they are data people. Combine that and now you create a pool in which every popular data tooling is written in python.
1
1d ago
[deleted]
1
u/Illustrious-Big-651 1d ago
And I guess it will break code silently if you and the libraries you are using or the libraries they are using do not handle that correctly.
Or does free threaded Python make all collection types per default threadsafe or did Python introduce some new threadsafe collection types to compesate for that?1
u/Captator 1d ago
The high uptake in scientific communities is what motivated the ML packages, not the other way round.
That high uptake was in part driven by Python being written as an accessible, multi-paradigm language to teach programming with at university. A dramatic upgrade on Java which preceded it, which itself (afaik) was a slightly industry-fad driven, but reasonable, movement away from using C or a lisp dialect.
1
u/Outside-Storage-1523 1d ago
Python is a glue language that calls libraries written in other highly performant languages such as C or Scala. You don’t really want to write from scratch.
0
u/Prior_Two_2818 1d ago
are your 20 years of experience in the room with us? or have you been the janitor of the tech firm?
28
u/Thwarting8139 1d ago edited 1d ago
This is a wild opinion for someone with 20 years of experience. How are you going to implement complex logic with SQL (as in UDFs), call third-party libraries, or package your code up with dependencies so other people can use it?
Also Python is extremely performant in practice. Any time you need higher performance than the language itself can provide, you just write something in C++, Java or Rust and then use it within your Python code - e.g. numpy, PySpark.