One architectural distinction I keep coming back to: people often confuse complexity with agency.

A system has multiple tools? -> “Use an agent.” OR It has five steps? -> “Definitely an agent.”

But neither of those things actually requires one. The more useful question is: who determines the execution path?

Consider an insurance assistant. If someone asks, “Am I eligible for this treatment?”, and the answer exists in internal policy documents, that's primarily a retrieval problem. And if they ask, “Check my claim status and tell me whether the rejected amount is covered under my policy.”

That might require more tools and more steps. But if those steps happen in a predictable order, is it still an agent ?

The interesting shift happens when the request is something like: “My claim was rejected. Find out why and tell me what I should do next.”

Now the path may not be known in advance. That's where an agent earns its complexity: when the system needs to help determine what to do next.

And Multi-agent can only consider it when there are genuinely distinct specialties, tools, or permission boundaries.

I think the common mistake is choosing “agent” as the starting point and then designing a problem around it. A better approach is to start with the responsibility:

Does the system need to know something? Decide something? Act? Verify the result?

Then add only the architecture required to support those responsibilities.

I mapped the complete e2e architectures and escalating examples out in more detail here, with visual breakdown: [https://youtu.be/kf5rSab4rcg\](https://youtu.be/kf5rSab4rcg)

For people building real AI systems: where do you draw the boundary between a complex workflow and an agent? Is dynamic tool selection alone enough for you, or do you require a more explicit decision loop before calling something an agent?

Hey everyone,

I’ve been working in the data science field for about 4 years now, but lately, I’ve been hitting a major confidence wall and could really use some unfiltered advice from those who have made it to top tech companies, secured high-paying remote roles, or are earning 30 LPA+.

Here is my honest reality check right now:

• The Foundation Gap: While I do day-to-day work, I often feel like my foundational concepts (in math, statistics, or core ML) have huge blind spots.
• The Portfolio Void: Outside of my professional projects, my personal portfolio is basically non-existent. I get overwhelmed trying to build end-to-end projects from scratch.
• The Confidence Trap: Because of the above, I feel severely underqualified to even apply to senior roles or top product companies, leading to a lot of hesitation.

If you are someone who has been in a similar spot or has successfully crossed into senior/well-paying data science roles, I would love to hear your perspective on a few things:

1. Breaking the Barrier: For those earning 30+ LPA or working remotely/at top tech firms, what actually moved the needle for you? Was it mastering core fundamentals, system design, cracking LeetCode, or deep domain expertise?
2. Fixing the Foundation: How did you go back and fix your weak foundational gaps while working a full-time job without burning out? Any specific resources or routines that worked?
3. The Interview Reality: What is the interview process really like at top companies right now? How heavy is the coding round compared to ML system design and statistics?
4. Portfolio Reality Check: Do personal projects actually matter as much as people say, or is it more about how you talk about your current work experience?

I’m ready to put in the work, but I feel like I'm running in circles right now. Any roadmap, hard truths, or advice you can spare would mean a lot. Thanks for reading!

Hello guys, i am a new learner in ds, kinda got a grasp of python, sql; numpy, pandas sqlearn and maths, but i want to look at a full project so that i would not stumble when dealing with real work at a job. So i want to look at kinda full prjoect that works with data bases, uses some coding and scikit learn for training models, perhaps neural networks and connects all of the stuff required in full project in itself. I just want to look at what i am working with so that i knew how to connect all the dots. Not sure how to frame my question, but if you understood it, could you tell me where to look for this kind of stuff? Because i am confident when it comes to part by part stuff, like working in sql on its own, writing code etc etc. but all of it combined seems like difficult to grasp for me

Technical data science skills are important, but working on real-world projects also requires strong non-technical abilities. Here are some key ones to focus on:

Problem Framing and Prioritization: Data teams often receive broad or vague questions, such as "Why are users dropping?" A strong data scientist knows how to narrow these questions down into something actionable and identify which questions matter most.

Business Context Awareness: Numbers rarely explain the full story on their own. A sudden dip in performance could be related to pricing changes, seasonality, supply issues, or other business factors. Understanding how the business operates helps data scientists interpret trends more accurately.

Working With Unclear Requirements: Many projects begin without clearly defined objectives or success metrics. Rather than waiting for complete clarity, data scientists can make informed assumptions, share preliminary results, and refine their approach based on feedback.

Decision-making Under Constraints: Time, data quality, and resources are often limited. Data scientists need to know when a quick estimate is more useful than a perfect model. For example, a simple trend analysis delivered today may be more valuable for planning than a complex model delivered too late.

Stakeholder Communication and Trust: Insights only create value when they are understood and trusted. Strong communication means presenting findings clearly, explaining limitations, and being transparent about assumptions rather than overstating precision.

Ownership Beyond Delivery: The work doesn't always end when a dashboard or report is delivered. Data scientists should also look at the decisions that follow, compare outcomes with expectations, and revisit their approach when the results don't match what was expected.

Staying Effective Under Pressure: Deadlines around product launches or review meetings can increase the likelihood of mistakes. Strong professionals slow down enough to check assumptions and avoid careless errors, especially during critical moments.

Adapting to Team Workflows: Different teams consume and use data differently. Some may need a concise summary, while others may prefer a detailed explanation. Adapting communication to the team's workflow can make it easier for insights to be understood and acted upon.

At that time, I wrote it as “nouns only”. When I checked, I found that there were 3,871 (-moq) verbs out of 31,993 entries. 😅

I also built a small Uzbek spell-checker based on this dataset and tested it with 200 synthetic typos:

🎯 Top-1 accuracy - 87.5%

🎯 Top-5 accuracy - 99.0%

⚡️ Average lookup - 2.6 ms

The main lesson is simple:

It is not a good idea to build a model on a dataset without checking the claims in it.

This time, I first audited it, then built the tool and measured the results.

🇺🇿 There is still a lot of work to be done for Uzbek NLP.

Hi everyone I have a Masters in Psychology and I absolutely love statistical analysis and research design

For the longest time I've wanted to be able to design real world projects that can create a huge impact from a psychological point of view

I'm good at identifying variables and hypothesising

But I have no clue as to how to begin with real code and python libraries to engineer analyse and present applicable insights from raw data

I would like to transition from therapeutics to behaviour science for businesses very soon

Can anyone please point out how that works and alif there are success stories from people starting as I am..

Thanks so much

Call for Papers: IC-AMMA 2026

The International Conference on Artificial Intelligence Methods and Modern Applications (IC-AMMA 2026) will take place on 12–13 December 2026 in Mangalore, India, in a hybrid format (in-person and online options available).

Promoted by Success Academic Associates for Intelligent Services (SAAS), the conference provides a multidisciplinary platform for researchers, faculty, and industry engineers.

Important Dates:
- Paper Submission Closes: 19 September 2026
- Notification of Acceptance: 10 October 2026
- Final Registration & Camera-Ready: 20 October 2026
- Conference Dates: 12–13 December 2026

Key Research Tracks:
1. Core AI Methods, ML, Deep Learning & Symbolic AI
2. AI & Data Analytics (Big Data Systems, Lifecycle, Risks)
3. AI Security, Privacy, Blockchain & Cybersecurity
4. AI for Healthcare & Smart Cities
5. AI for Education & Personalized Learning
6. AI-driven Management & Decision Support Systems
7. Multimodal & Cross-Domain AI Systems
8. Generative, Agentic & Quantum AI
9. AI Innovations & Edge AI
10. Future Horizons & Machine Autonomy

Publication:
All accepted and registered papers will be published in a SCOPUS-indexed publication (Elsevier Computer Science Procedia, permission awaited).

Submission & Details:
Submission is via the Microsoft CMT portal. Links to the submission portal and main conference website are provided in the comments below.

When people fine-tune their own local models, the model setup usually gets most of the attention. But in practice, a lot of the work seems to be on the data side.

If you are training on business data, you may need to bring in support tickets, internal docs, product specs, chat logs, code, policies, CRM notes, or domain QA pairs. And it usually does not work perfectly on the first run. Some samples are noisy, some are redundant, some domains overpower others, and some “bad-looking” examples are actually hard but useful.

One direction I have been thinking about is making the data strategy dynamic during training.

Dynamic selection means periodically choosing which samples should enter the next training window, using signals like loss, delta loss, gradient similarity, or external scores.

Dynamic mixing means adjusting the ratio between data sources during training, instead of fixing one static mixture before the run.

Dynamic weighting means keeping the sample in training, but changing how much its loss contributes to the gradient update. This is useful when you do not want to hard-drop uncertain samples.

This is the current direction in OpenDCAI/DataFlex: adding data selection, data mixing, and data weighting controls on top of the training loop.

For people here who fine-tune local models, how much time and compute do you usually spend on data preparation compared with the actual training run?

In a previous post I shared a structured approach to projects (with a list of 99 project ideas for inspiration). This is a follow-up post, which goes through exactly how to set up a data science project following that approach. The code is also available for download.

The focus is not on how to do the EDA and model development, but rather on how to set up the project so that you can deliver value right away.

The example project is deliberately very simple, to better focus on the structure and setup. We use test-driven development throughout, a technique I've found very helpful for preventing bugs and making it easy and painless to make changes. The idea is that you can take this very simple structure and iterate on the actual model and data, and still have something working at the end of the day.

Of course you can always ask AI to set this up for you, but, I still think there's value in learning how to do it. I have also found that having tests makes doing it with AI much faster and smoother.

Hey Everyone I am working on prescription and doctor dataset right now.The idea is to built a churn risk model.

The issue: I have two cases that look almost identical to the model, but shouldn't be treated the same:

Doctor A has been climbing steadily for two years starts small, ends up writing a lot. Right now, this month, they're near their highest ever, because they've genuinely been growing.

Doctor B used to write a lot, but has been sliding downward for months. Right now, this month, they're also unusually high compared to their recent low months maybe they just had one slightly better month in the middle of an overall decline.

What the model is doing wrong: across almost all doctors in our data, there's a common pattern whenever someone's number is unusually high this month, it's usually a bit lower next month, just because most "unusually high" months are one-time spikes that settle back down. That's true most of the time.

But the model applies this same rule to every doctor whose number is currently high — including Doctor A, who isn't having a fluke month, they're genuinely growing. So my "who's about to decline" list keeps getting filled with doctors who are simply doing well right now — because "currently high" is the one thing they all share, not that they're actually declining.

What I've tried so far, to fix it:

• Switched from weekly to monthly data (to reduce noise) didn't fix it
• Compared 4 different model types (linear, ridge, random forest, gradient boosting) — all 4 show the exact same bias
• Rebuilt the trend line to use only the last 9 months instead of the full 2 years — didn't fix it
• Added the weekly short-term trend back in, like you described (weeks within the recent month) didn't fix it
• Removed the features causing the biggest pull toward "predict a drop" entirely, to force the model to rely on trend instead the model just found other features to reproduce the exact same wrong prediction

So is this one bad feature or one bad model choice ?? I've tested that directly, several ways, and the bias holds regardless.

What I want to ask you:

1. Is this the kind of thing that genuinely needs more historical data than 2 years to fix (i.e., is 2 years just not enough for the model to learn "normal high point" vs. "real decline" apart)?
2. Or is there a different way to frame the target/features you'd suggest something specifically designed to separate a real trend break from ordinary noise, rather than predicting the raw next-month number?

I work in a regulated area so I'm going to be vague on the details of the data I work with, but to provide structural context. The data I'm working with is entries collected over time associated to a known ID. Basically I want to analyze all the events that have occurred to a specific ID and use the types of events and when they occurred to predict another value.

What I'm struggling with is how to structure the data for this. Originally I tried to pivot out the events into columns but there's multiple associates data points with each even that can't simply be stored nicely in the column. How should I think about structuring this data?

I'm currently student and am building a portfolio so i created this library which im calling Async batch faker. So if you have used standard faker you already know that "Faker" is good at generating a single row but generating a bit more takes forever because it operates on pythons standard for loop.
So what i made async batch faker do is that it uses numpy and asyncio to calculate mathematical data and string concatenations instantly in C. It bypasses Python loops to blast hundreds of thousands of rows of localized data in seconds

Benchmark:
Standard Faker: ~348 seconds
Async Batch Faker: ~8.4 seconds (41x faster)

GitHub: [https://github.com/haiderkashan/fast_faker_poc\]
PyPI: pip install async-batch-faker

I'm just a rookie must have made mistakes would love to hear feedbacks

Hi there,

I've created a video here where I explain how generalized linear models work.

I hope some of you find it useful and as always, feedback is very welcome! :)

Hello,

I’m an engineer and already work with data on a daily basis. I have a basic background in SQL and Python, as well as some experience with statistical tools, and I recently enrolled in a Lean Six Sigma Green Belt training.

I’m now looking to improve my data analysis skills, particularly in SQL and Python, but I’m also interested in developing a stronger foundation in how to structure analytical problems, choose an appropriate approach, explore data, and communicate insights effectively.

My goal is to start with a solid, structured foundation and then move on to more advanced, topic-specific training.

I’ve been considering the IBM Data Analyst Professional Certificate and the Google Advanced Data Analytics Professional Certificate, but I’m also open to other recommendations.

For those with experience in data analytics or who have completed these courses which one would you recommend? Is there another certificate or course you think would be a better fit for someone with some existing practical experience?

I’m particularly interested in courses that combine practical work with strong analytical foundations, rather than focusing mainly on the certificate itself.

Any recommendations or experiences?

Hi everyone! 👋

I’m a Class 11 student from India learning Data Science. I recently completed and deployed an end-to-end Salary Prediction project using Python, SQL, Pandas, data visualization, and Machine Learning.

🌐 Live Demo: https://data-science-projects-fdkdsvuf5rvtywpwby35py.streamlit.app/

📂 Project: https://github.com/lakshay-OG-DS/data-science-projects/blob/main/Salary\\_prediction.ipynb

I’d really appreciate honest feedback from Data Scientists, Data Analysts, and ML Engineers.

What is the one biggest thing I should improve to make this project more internship-ready?

Even one small suggestion would mean a lot.

I have been working as a network engineer in one of WITCH companies . But to be honest the rotational shift overwork with no growth is killing me slowly.

I also do not see much callbacks or opportunities of the field.

So if I were to make a move to data science how should I approach. And what resources should I follow both free and paid..

Target roles are mainly data science.. also I am not aware of the difference of ML engineer and data scientist

I am a junior torn between two mindset, should we

1. Start with a business problem/ use case first
but I often run into data limitations after diving deep, like if the data is a suitable proxy of something or there are missing values

2. Explore the data freely and hunt interesting patterns
but I am always confused where I should start and not being lost on the way of that) (but I am always confused where I should start and end up spinning around without a clear direction

Example

Say a company gives you customer purchase history and asks you to "find something useful."

We better immediately frame it around a specific use case (e.g., next-best-offer) and engineer toward that?

Or spend time clustering, looking for seasonality, correlations, or weird segments first, and then figure out what business value those patterns might have?

Hey everyone,

I have an upcoming Data Scientist interview with an ML system design round. I asked the recruiter what to expect and they said it’ll be high-level ML system design, not traditional SWE/low-level system design.

They mentioned focusing on things like:

• Problem framing
• Data/model considerations
• Evaluation
• Productionization/deployment
• Monitoring
• Tradeoffs when designing ML systems

Basically, it sounds like I’ll be given a real-world ML problem and have to explain how I’d approach it end-to-end.

This is my first dedicated ML system design interview, so I’m not really sure how deep I should prepare.

For people who’ve done these interviews:

How did you prepare? What resources did you use? What kind of questions were you asked?

Also, how deep do they usually expect you to go into things like feature stores, model serving, APIs, streaming, retraining, etc.?

Any good resources, YouTube playlists, GitHub repos, books, or example questions would be really appreciated.

Thanks!

In last week’s post we showed that a Python for-loop works through the Iterator Protocol.

This same protocol also enables values to be produced lazily.

In this new Lazy Evalution Example we show one sink reading values from five different sources: - source1: eagerly returns all values by list - source2: lazily by generator function using yield - source3: eagerly by list comprehension - source4: lazily by generator expression - source5: lazily by iterator protocol

The same for-loop consumes all five sources. The eager sources produce all values before the sink starts consuming them. In the lazy sources producer and consumer take turns:

• produce → consume → produce → consume → …

where each value is produced only when the for-loop requests it.

Generator functions and generator expressions are concise, readable ways to create lazy iterables. The final source makes their underlying mechanism explicit: - iter() obtains an iterator. - next() requests the next value. - StopIteration signals that no values remain.

This clearly shows that generators achieve lazy iteration by implementing Python’s Iterator Protocol.

See other 𝐦𝐞𝐦𝐨𝐫𝐲_𝐠𝐫𝐚𝐩𝐡 visualizations.

Hey I'm 22 completed ug in maths now pg in data science first year on going . I'm blind where I could start working on data science related things, clg providing full maths concepts and program now I'm learning python,R programming, ig it's not enough to enter into job market , please give a suggestion where I could learn more about data science projects and how to work on it .