Partner sends us a monthly export. Different customer ID scheme to ours, no mapping table, no docs, and whoever built it on their side left last year.

Right now I fuzzy match on lower(trim(email)) plus surname and eyeball a sample before it goes anywhere. It works well enough that nobody's complained, which is not the same as it being right. I have no idea what my false match rate is.

The part that bothers me is that a bad join doesn't announce itself. A wrong customer count looks identical to a right one.

So what do people actually do here. Do you block on something cheap first to cut the comparison space, compute a score and hold back anything under a threshold, or just accept a fuzzy match and monitor downstream? And if you threshold, where did the number come from?

Hello fellow traders,

I have been trading for about 6 years now and been through the highs and lows coincidently like on the charts haha. I have done crazy numbers and then lost everything many times. Through my most hardest times, i noticed the truth behind trading. Success in the space is not determined by your efforts or study, but rather in identifying RELEVANT patterns then taking advantage of RELEVANT data.

Men lie, women lie, but numbers don’t.

My fascination with data and numbers has grown immensely through my period of isolation. I have been able to crack sports finals EXACT scores using data (knicks, spain wc, etc). The correlations I can find/show are truly mind shattering. I am willing to share how i see numbers to whoever is interested.

I’ve decided to use this power of mine to create a platform for traders that solves every problem I had in my own trading as well as more universal problems (course selling larpers, unappreciated low income traders, emotional trading, etc).

Trading has ruined my life for the most part and the suffering that came with it has been the fuel in creating this platform. As the niche continues to grow, the real will be filtered out from the dream selling methodologies out there. I want to be a part of this movement so no other child bears the emotional pressure or confusion that comes with trading.

Trading will always be a temptation, even for those who ARENT supposed to be trading. I want to at least have my platform as space for analytical and shared decision making at a gamified level. I also want to give credit only where credit is due. This is not just another “copy trader” platform. This is quite literally the future.

I truly believe I have created something unique and worth all my efforts.

I am currently looking for HONEST strategic partners and investors in the same space. I do not want to wait any longer and would appreciate the communities advice on how/where to find investors. I am more comfortable with sharing in depth details in a live meeting in NJ or NYC.

Currently there is no one using the platform and I need money for devs. The start up is done but non functional at a multi user level.

Any advice is truly appreciated. Thank you.

hey I have seeing report from CBRE on indian realestate

i was just curious how are they scrapping public data and analysing it can I also do it myself?

I compiled a dataset tracking global FDA enforcement actions, inspection records, and regulatory raids across the food and drug sectors.

Key Fields: Inspection dates, target facility names, geographic locations, violation categories, enforcement actions, and compliance status.
Useful for EDA and ananlysis

I was developing different dataset for the last 1-2 months lately, but unfortunately I realized it is so hard to earn medals from creating dataset. Even if you got 500+ download, sometimes I cannot get any upvote. I do not know how it works exactly.

So I realized taking some advice would really matter right now. Is there anyone can help me with it. You can take a look at my profile and dataset, any feedback matters. Thanks!

https://www.kaggle.com/farukece

I’m doing a research which involves chat messages from teams, slack, google chat etc. For the software project i need to train a dataset. So dataset should be related to developer chat messages/logs of a specific project. How can i find the dataset?

I've been experimenting with how much interpretation to build directly into a chart.

Take a time series: the data might be clear on its own, but the actual story is often a crossover, a peak, a reversal or an unusual divergence. Highlighting those moments can make the chart much faster to understand. Highlight too much, though, and you're basically telling the reader what to see.

I've found the same trade-off with labels and explanatory text. More context isn't necessarily better context.

How do you approach this? Do you prefer charts that actively surface the interesting parts of the data, or keep the visualization relatively neutral and let the reader interpret it?

I was hired as a digital transformation consultant by a construction company with 10+ active construction sites the Finance director wanted a centralized view of labor deployment and material consumption but getting overworked site teams to open a laptop and fill the form or fill excel sheet was impossible so I built a work around site engineers have to just dump all the information in an email --> AI converts raw text into google sheets and sends summary email to Project manager --> a code to save the data in a database so old data don't get lost or deleted --> private streamlit dashboard for centralized visibility. Finance director got the visibility, no extra work of site engineer and a full functioning dashboard to see what is happening on ground without calling 10 different project managers and site engineers.

I'm curious how other construction companies handle this.

Do your site teams actually fill out digital daily reports consistently, or does someone in the office end up chasing everyone for the information?

This dataset represents real-world, masked, and anonymized customer data from the BFSI (Banking, Financial Services, and Insurance) domain. It captures a variety of customer demographics, financial indicators, and product interaction metrics collected during a loan application process.

To comply with strict data privacy laws and protect user identity, all sensitive personal identifiable information (PII) has been securely encrypted or masked. However, the underlying statistical relationships, distributions, and patterns remain fully intact, making this an ideal playground for building robust classification models.

Anything about some of the less obvious challenges of dimensional modeling, the decisions around grain, dimensions, facts, slowly changing dimensions and how things get messy in real-world projects.

Does anyone have any advice on where to go from here? The algorithm simply goes through all feature-attribute combinations in a dataset and scores it on purity of the target column. It's very slow, as the number of combinations increases exponentially with the number of input features, and so can only work with 3-4 features at a time. But my goal is to identify <=5 important features with a linear model (statsmodels or PyGam), and then plug in those features to this algorithm to see which subset of the market might be smart to target.

Here's an example with some charts, on the iris dataset.

https://colab.research.google.com/drive/1w3VpDlDATdHwtSk-RmQPMj36cgOfthz1#scrollTo=ZOqeYiMRalj7&uniqifier=1

New to algorithms, so any advice would be welcome. Thanks in advance.

I’m working on an idea around **data quality** and I want to understand how this works in real projects.

When a data batch comes into your pipeline and you find some problems, how do you decide what to do?

For example:

* Some records have missing values * The schema has changed * There are duplicate records * Some values look wrong * Only a small percentage of records are bad * The data looks technically correct but something seems wrong from the business side

Do you normally:

**Accept the batch, fix it, quarantine the bad records, or reject the whole batch?**

And who makes that decision — an automated rule, the data engineer, business team, or someone else?

I’m especially interested in cases where **you don't have enough information to make a clear decision**.

If you work with data pipelines in production, I’d really appreciate hearing about:

* What kind of data-quality problems you face most often? * What do you currently do when they happen? * What part of the process is still manual? * What would you like to automate?

I’m trying to understand the **real problems engineers face before building anything**.

Thanks in advance for sharing your experience!

Nov 4th and 5th [https://www.dataoutpost.ai/\](https://www.dataoutpost.ai/)

Seems like a pretty small conference, \~250 people. In the past years at least, you have choices for the first day (workshops) then for the second day, everyone is hearing the same talks.

If you've been, is it worth going? I'm also not local, so there's a bit more cost involved. My company might not be able to sponsor me going.

I read an interesting piece recently that made a distinction I think the BI industry needs to talk about, data portability ≠ sending someone a file.

In most organizations, “portable data” means exporting a CSV, PDF, Excel workbook, or maybe dropping something in object storage and calling it done.

But what actually traveled?

Usually not much:
- not the business logic
- not the semantic model
- not the drill paths
- not the interactive analytics layer
- not the governance model
- not the ability to work with it meaningfully offline

What moved was often just a snapshot.

That’s where I think BI has a blind spot. We’ve built an ecosystem where intelligence is tightly coupled to platforms like dashboards, cloud runtimes, query engines, licenses, network access, identity systems, etc. The moment information leaves that stack, it often collapses into static content. Is it portable? Technically. Is it useful? In a way.

Real portability should mean:
* intelligence that travels with the data
* embedded context and business rules
* secure scoped distribution
* offline usability
* interoperability without losing dimensionality
* reduced dependency on always-on compute infrastructure

As cloud costs rise, AI inference costs grow, and governance gets tighter, I suspect the future of BI may shift from “always connected dashboards” toward portable intelligence artifacts self-contained analytic experiences that can move securely between systems, organizations, and users without losing capability.

Curious how others here see it:

When you say “data portability” in BI what should actually be portable, just data,Or the intelligence layer too?

Hey guys, I have a dataset of around 180k adult stool images, growing at a 350+ images per day, and I’m trying to better understand the “right” way to bring monetary value from it .

The app was launched about 2 years ago.

I am very transparent with the use and licensing of the data that flows through the app.

The dataset consist of the image and a lot of useful stool metadata (Bristol Type, blood, mucus, color, etc..), anonymized user habits and stool related specifics (time of day, region, meal, allergies, weight, gender, height, activity, etc..)

I am here seeking advice on how such data can be useful for business growth. I am finding very little luck in this part of the B2B and I initially thought that becoming the Google of Poops would be my primary moat.

Thanks in advance

I'm currently taking a Data Analysis course, and I need to complete a final project. The assignment requires me to choose a topic, formulate one or more hypotheses, and build a thesis/research structure around it.

One of the requirements is that there must already be accessible data available for analysis, so I can't pick something that would require collecting a completely new dataset because the core of the project is not in that stage. Also, what I'll be researching needs to be answering or try to solve an existing problem.

My problem is that I can't come up with a topic that feels genuinely interesting or motivating enough to spend several weeks working on. Most ideas I've found online seem either too generic (social media usage, movie ratings, etc.) or too complex for a course project.

Do you have any suggestions for topics that have publicly available datasets, and still are interesting and have potential for drawing conclusions?

I'm open to almost any field (health, economics, business, technology, environment, etc.)

I'd love to hear about projects you've enjoyed working on or topics that sparked your curiosity - thanks in advance!

I pulled all 1.3 million federal contract records from the Government of

Canada's open data and looked at one question: who's holding contracts that

are about to come up for renewal?

A few things surprised me.

26,240 federal contracts are currently live, worth $125B. Of the ones that

report a bidder count, 75% drew one bid or none.

And roughly 63 contracts cross into the 12-month planning window every single

week — about $19B of contract value a year.

That last number is the interesting one. Agencies typically start recompete

planning 12–18 months before a contract ends. So the moment a contract crosses

that line is the moment it becomes worth a conversation. It's all public, but

nobody was watching for it.

So I built something that does. It's free, it updates itself monthly, and it

shows you which contracts are expiring, who holds them, what they're worth,

and how contested they were last time.

No signup needed to browse it. Link in the comments.

If you bid on federal work, I'd genuinely like to know whether this is useful

or whether I've missed something obvious.

We've posted a few times about where agriculture sample data tends to break, so pulling it together into one list for anyone mid-evaluation right now.

Before you sign, get answers on:

• Field context capture: required fields at registration, not optional notes added later.
• Sample type flexibility: independent schemas for soil, seed, plant tissue, water, and grain, not one template stretched across all of them.
• Multi-season continuity: lineage that connects across growing seasons without a manual report request.
• Chain of custody: timestamped, role-attributed logs at every handoff, not a paper trail reconstructed after the fact.
• Self-service reporting: whether your research team can build a cross-season, cross-site view without IT involvement.

The labs that get the most out of whatever system they land on are the ones that define their data structure (sample types, required fields, season linkage) before the first sample gets registered.

If you're in the middle of an evaluation and want to compare notes on how a specific system handles any of this, drop it below and someone might be able to help.

Hey yall,

I just started my new role as a junior data analyst 🥹. I am learning the business needs. I work for a no profit org that provides legal aid. Cases and task are billed to the respected grant program. What we are running into is, while closing the books normally case workers and lawyer over bill to one program vs another because no can visually see what is currently in “encumbrance for that month.

I figured building a real time grant funding system using power bi, postgreDQL and RESTapi from the platform that we input the data in. I am open other ideas that could be easier.

Thanks in advance

Claude and other LLM models can be so frustrating. I asked it to find patterns across five campaigns regarding what a person buys and sells, and under which conditions, but it couldn't do it. It just kept saying it was impossible. I literally had to explain every single detail to it like it was a child, even though the data was cleanly split across five CSV files.

Worse, companies are stopping the hiring of junior engineers because they think these tools can replace them. They are going to cause a massive talent shortage, and then these dumb models won't be able to do anything without skilled people to guide them.

Hello everyone,

The core principles of Data Warehousing and Consumer Data Products, establishing the foundation for a 365-day technical log documenting pipeline architecture, schema design, and engineering tradeoffs.

Technical Breakdown:

Relational Database Fundamentals: SQL query execution mechanics, indexing strategies, and relational constraints.

OLTP vs. OLAP Paradigms: Comparative tradeoffs between transactional database normalization and analytical denormalization.

Ingestion Foundations: High-level mechanics of staging layers, raw event ingestion, and downstream transformation logic.

Current Focus:

Pivoting from relational database mechanics to dimensional modeling paradigms—specifically Kimball methodology, star/snowflake schemas, and event-driven data product architectures.

For engineers working with production systems: What are the primary pitfalls to avoid when transitioning from standard relational models to analytical warehouse schemas?

As part of my thesis for my MSc I want to examine the year-by-year social media following for each of the Big Six (Arsenal, Chelsea, Liverpool, Manchester City, Manchester United, Tottenham) clubs from 2019/20 to 2025.

However, I can't find it at all and in my eyes this should be reasonably available data. I've already tried Socialblade and Statista. Any help/pointers would be much appreciated!