Solo dev. Just released everything from a weeks-long ML project and wanted to share both the release and a licensing gotcha that might save someone else the headache.

What's open:

• Code: Apache-2.0, on GitHub. Non-autoregressive decoders that turn sentence embeddings into knowledge-graph triples (for GraphRAG, agent memory, that kind of thing).
• Weights: 11 trained checkpoints, free on Hugging Face.
• The full test suite (113 tests, runs offline).
• The changelog documents negative results too — every approach that failed and why. I think hiding the failures makes releases less useful, so they're all in there: the loss function that made things worse, the LLM-distillation attempt that collapsed, the char-level generator that scored 0.006.
• Training recipes are reproducible: same splits, same seeds, documented protocol.

The licensing lesson: my decoder heads are trained from scratch, so Apache-2.0 was easy. But they consume embeddings from Meta's SONAR encoder — and SONAR's weights are CC-BY-NC 4.0 even though its code is MIT. Which means: my Apache-licensed decoders are useless commercially without a non-commercial encoder running upstream. The NC restriction attaches at runtime, not at my artifact level. I only fully worked this through after publishing, wrote an internal due-diligence doc, and the fix is on the roadmap: migrating to BGE-M3 (MIT-licensed weights, same embedding dimension, so the architecture doesn't even change).

If you're building on top of any "open" model: check the weights license separately from the code license. They differ more often than you'd think.

Repo: https://github.com/DeliVali/cogito-estella

Questions for this community:

1. For those who maintain ML projects: do you publish negative results/failed experiments, or just the wins? I'd like to know if anyone else finds this valuable or if I'm just cluttering my changelog.
2. How do you handle the mixed-license situation (permissive code, NC weights upstream) in your docs? I disclosed it in README + release notes + model card, but curious what the standard is.
3. Solo maintainer here — what's the one thing that made your project contributor-friendly early on?

Before claiming SOTA, check if the "magic" of your graph topology disappears when you simply add edge counts to a baseline MLP. GNNs often degenerate into basic MLPs when node degrees correlate heavily with tabular features like transaction volumes. The model simply learns the feature marginal distributions rather than the graph topology. If the graph structure doesn't provide independent signal, it's redundant. High AUCs on such datasets usually indicate tabular leakage, not structural learning.

In the synthfin-aml V9.1 dataset update, we neutralized the tabular distributions to isolate the structural signal and eliminate this leakage. As a result, standard tabular baselines drop from 0.99 PR-AUC to 0.31 PR-AUC. This decline is expected—it confirms the removal of spurious correlations, forcing models to rely entirely on graph topology.

We submitted this benchmark upstream to PyTorch Geometric (PR #10774) to establish a stricter evaluation standard.

Curious if anyone has found reliable ways to prevent feature marginals from dominating structural signal in production.

Link: PyTorch Geometric PR #10774

PyTorch Conference North America is just around the corner & I'd love to have you join us in San Jose, CA from October 20-21. Ticket prices go up in 1 week.

This year’s conference is going to be EPIC.

• Stellar keynotes
• 150+ sessions spanning foundational concepts and core framework work to training, inference, applications, kernel engineering, and responsible AI
• 140+ poster presentations
• BoFs
• Meet the developers
• Flare party
• AI community bash
• +more.

Sign up by September 4th to save $200. Register now.

I see a lot of people asking how to get PyTorch running on AMD hardware without CUDA. I put together a detailed guide that covers ROCm installation, GPU detection, performance tuning, and troubleshooting. It’s based on my own experience getting it stable on a 7900 XTX. Hope it helps someone.

https://interconnectd.com/forum/thread/248/pytorch-on-amd-gpus-the-complete-rocm-setup-tuning-guide/

Hi, I'm learning PyTorch from 'AI and ML for Coders in PyTorch'

I ran the example code below on google colab.
And I got 55% validation accuracy at epoch 10.
But, the book says it gets 87% validation accuracy at epoch 10.

Why is there a large gap between the book's and mine?

import urllib.request
import zipfile


url = "https://storage.googleapis.com/learning-datasets/horse-or-human.zip"
file_name = "horse-or-human.zip"
training_dir = 'horse-or-human/training/'
urllib.request.urlretrieve(url, file_name)


zip_ref = zipfile.ZipFile(file_name, 'r')
zip_ref.extractall(training_dir)
zip_ref.close()


url = "https://storage.googleapis.com/learning-datasets/validation-horse-or-human.zip"
file_name = "validation-horse-or-human.zip"
validation_dir = 'horse-or-human/validation/'
urllib.request.urlretrieve(url, file_name)


zip_ref = zipfile.ZipFile(file_name, 'r')
zip_ref.extractall(validation_dir)
zip_ref.close()



from torchvision import datasets, transforms
from torch.utils.data import DataLoader


# Define transformations
train_transform = transforms.Compose([
    transforms.Resize((150,150)),
    transforms.RandomHorizontalFlip(),
    transforms.RandomRotation(20),
    transforms.RandomAffine(
        degrees=0,  # No rotation
        translate=(0.2, 0.2),  # Translate up to 20% vertically and horizontally
        scale=(0.8, 1.2),  # Zoom in or out by 20%
        shear=20,  # Shear by up to 20 degrees
    ),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.5, 0.5, 0.5], std=[0.5, 0.5, 0.5]),
])



# Load the datasets
train_dataset = datasets.ImageFolder(root=training_dir, transform=train_transform)
val_dataset = datasets.ImageFolder(root=validation_dir, transform=train_transform)


# Data loaders
train_loader = DataLoader(train_dataset, batch_size=32, shuffle=True)
val_loader = DataLoader(val_dataset, batch_size=32, shuffle=True)




import torch
import torch.nn as nn
import torch.nn.functional as F


class HorsesHumansCNN(nn.Module):
    def __init__(self):
        super(HorsesHumansCNN, self).__init__()
        self.conv1 = nn.Conv2d(3, 16, kernel_size=3, padding=1)
        self.conv2 = nn.Conv2d(16, 32, kernel_size=3, padding=1)
        self.conv3 = nn.Conv2d(32, 64, kernel_size=3, padding=1)
        self.pool = nn.MaxPool2d(2, 2)
        self.fc1 = nn.Linear(64 * 18 * 18, 512)
        self.drop = nn.Dropout(0.25)
        self.fc2 = nn.Linear(512, 1)  # Only 1 output neuron for binary classification


    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = self.pool(F.relu(self.conv3(x)))
        x = x.view(-1, 64 * 18 * 18)
        x = F.relu(self.fc1(x))
        x = self.drop(x)
        x = self.fc2(x)
        x = torch.sigmoid(x)  # Use sigmoid to output probabilities
        return x





import torch.optim as optim


device = torch.device("cuda" if torch.cuda.is_available() else "cpu")


model = HorsesHumansCNN().to(device)
criterion = nn.BCELoss()
optimizer = optim.Adam(model.parameters(), lr=0.001)


def train_model(num_epochs):
    for epoch in range(num_epochs):
        model.train()
        running_loss = 0.0
        for images, labels in train_loader:
            images, labels = images.to(device), labels.to(device).float()  # Convert labels to float
            optimizer.zero_grad()
            outputs = model(images).view(-1)  # Flatten outputs to match label shape
            loss = criterion(outputs, labels)
            loss.backward()
            optimizer.step()
            running_loss += loss.item()


        print(f'Epoch {epoch + 1}, Loss: {running_loss / len(train_loader)}')


        # Evaluate on training set
        model.eval()
        with torch.no_grad():
            correct = 0
            total = 0
            for images, labels in train_loader:
                images, labels = images.to(device), labels.to(device).float()
                outputs = model(images).view(-1)
                predicted = outputs > 0.5  # Threshold predictions
                total += labels.size(0)
                correct += (predicted == labels).sum().item()
            print(f'Training Set Accuracy: {100 * correct / total}%')


        # Evaluate on validation set
        model.eval()
        with torch.no_grad():
            correct = 0
            total = 0
            for images, labels in val_loader:
                images, labels = images.to(device), labels.to(device).float()
                outputs = model(images).view(-1)
                predicted = outputs > 0.5  # Threshold predictions
                total += labels.size(0)
                correct += (predicted == labels).sum().item()
            print(f'Validation Set Accuracy: {100 * correct / total}%')
train_model(15)



model.eval()
with torch.no_grad():
    correct = 0
    total = 0
    for images, labels in val_loader:
        images, labels = images.to(device), labels.to(device).float()
        outputs = model(images).view(-1)
        predicted = outputs > 0.5  # Threshold predictions
        total += labels.size(0)
        correct += (predicted == labels).sum().item()
        print(outputs)
        print(labels)
    print(f'Validation Accuracy: {100 * correct / total}%')

Preventing Grid Collapsing in Neural PDE Solvers: A lightweight PyTorch Log-Barrier Loss for 2D Transformation Matrices

Hi everyone!

When training neural operators (like FNOs) on non-convex physical domains, spatial grid points can overlap during optimization (\det J \le 0).

To fix this topology failure, I wrote a lightweight PyTorch module `JacobianBarrierLoss` that enforces strict positive volume elements during backpropagation using analytical 2x2 determinants directly executed on GPU.

```python

import torch

import torch.nn as nn

class JacobianBarrierLoss(nn.Module):

def __init__(self, eps=1e-4, alpha=1.0):

super().__init__()

self.eps = eps

self.alpha = alpha

def forward(self, J):

# Fast 2x2 analytical determinant (ad - bc) avoiding torch.linalg.det overhead

det_J = J[..., 0, 0] * J[..., 1, 1] - J[..., 0, 1] * J[..., 1, 0]

safe_det = torch.clamp(det_J, min=self.eps)

barrier_loss = -torch.log(safe_det).mean()

return self.alpha * barrier_loss

We integrated this into DIF-FNO to achieve diffeomorphism on complex geometries (Star/L-Shape/Annulus) without grid folding.

Repository GitHub: https://github.com/GiovanniDagnese-paper/DIF-FNO

Preprint & DOI: https://doi.org/10.5281/zenodo.22071926

Feedback on the PyTorch implementation and repository architecture is welcome

Fine-tuning a 1B sparse MoE (305M active, custom trained from scratch, ~100B tokens). Every narrow SFT run catastrophically overwrites existing behavior within 5–10 steps, regardless of what the data contains.

Seven runs now, same signature: whatever the recent batch over-represents gets installed near-perfectly, everything else degrades. A 2,000-row corpus at 127-token median taught a new capability 0% → 98% in five steps while unrelated call-formatting went from 1.4% error to 31%. Pure pretraining replay with no task data at all also degraded task behavior. Cold-init and verified true-resume of optimizer state both degrade, resume slightly worse.

Config: ~1M tokens/step, 60/40 replay/task, lr_mult 0.05 flat, Muon + AdamW, seq_len 4096.

Is this normal for small MoEs, or a sign of something wrong? Is 1M tokens/step simply too large a batch to fine-tune this gently? Would LoRA or a much lower LR change the picture, or is dilution into a large balanced mixture the only real fix?

Bug description

Testing from a given checkpoint leads to logging the epoch number of the last checkpoint instead of the checkpoint specified:

trainer = Trainer(..., max_epochs=10)
lightning_module = MyLightningModule(...)
datamodule = MyDatamodule()

trainer.fit(lightning_module , datamodule=datamodule)

trainer.test(lightning_module , datamodule=datamodule, ckpt_path="last")     # <-- ok: logs correct epoch and step
ckpt_path="/.../checkpoints/epoch=2-step=396.ckpt"
trainer.test(lightning_module , datamodule=datamodule, ckpt_path=ckpt_path)  # <-- incorrect: logs last epoch and step

The second test logs epoch 10 instead of epoch 2. Similarly, the step number of the second test is incorrect.

What version are you seeing the problem on?

Would anyone be interested in helping me develop some of my code to help get me started on making neural networks? I am wanting to make a simple NLP encoder decoder model for seq2seq artificial language translation but I cannot seem to get any traction. If I show you some of what I have already, can you push me in the right direction? All I need is something more human than chatGPT to push me in the right direction. Maybe I can put it in a google colab notebook and you can help me get something running? I have tried looking through lots of stuff and cannot find out what I’m doing wrong.

Released HyperSAE, a PyTorch library for training Sparse Autoencoders with hyperbolic weight regularization.

GitHub: https://github.com/vishal-dehurdle/hypersae Install: pip install hypersae

Design decisions:

1. The forward pass is standard Euclidean linear algebra. No custom CUDA kernels, no Riemannian optimizers in the hot path. This means zero inference overhead and full compatibility with torch.compile, FSDP, and existing steering pipelines.
2. Hyperbolic geometry is applied only to dictionary weights during training via a Poincaré ball projection + entailment cone loss. This regularizes the weight manifold without touching activations.
3. Single-class trainer interface:from hypersae import HyperSAE, HyperSAETrainersae = HyperSAE(d_model=2304, dict_size=16384) trainer = HyperSAETrainer(model=sae, lr=1e-3) metrics = trainer.train_step(batch)
4. TriPartite loss function combines reconstruction MSE, L1 sparsity, and Poincaré entailment with configurable coefficients:from hypersae import TriPartiteLoss loss_fn = TriPartiteLoss( l1_coeff=0.005, entail_coeff=0.01 )
5. Co-activation queue tracks feature co-firing patterns for hierarchy discovery without gradient overhead.

Benchmarked on Gemma-2-2B Layer 13 (20M tokens, L4 GPU): reconstruction MSE drops 9.8%, dead latents drop from 3.8% to 0.2%.

Paper: https://vishalvermalabs.com/papers/empirical-validation-hypersae-poincare-geometry/

Feedback on the API design welcome.

PyTorchCon NA 2026 (October 20-21, 2026 in San Jose, CA) keynote lineup is live: https://events.linuxfoundation.org/pytorch-conference-north-america/program/keynote-speakers/

Full Schedule: https://events.linuxfoundation.org/pytorch-conference-north-america/program/schedule/

Tickets available at a discount through September 4th: https://events.linuxfoundation.org/pytorch-conference-north-america/register/

Hey folks!

Did you ever wrapped model(x) in time.perf_counter() and gotten numbers that make no sense?

I realized it's a common enough trap and wrote a detailed write up here:

https://medium.com/traceopt/two-clocks-one-training-step-how-traceml-measures-pytorch-performance-357bc8e28dc7

TL;DR:

CUDA runs async. model(x) just enqueues kernels and returns, so a perf_counter() bracket around it measures how long Python took to queue the work, but not how long the GPU took to run it. The pending GPU time gets charged to whatever blocks next.

The tried the textbook fix, torch.cuda.synchronize() before each reading, which gives you accurate numbers but entirely about a different run.

Every sync becomes a stall, and it serializes exactly the CPU/GPU overlap you were trying to measure.

If one tires CUDA events (start.record() / end.record() / elapsed_time), it may fix both: the GPU stamps the markers as it passes, and you read them later with a non-blocking query() so nothing ever waits.

But i realized "CUDA events everywhere" is also wrong.

DataLoader next() is CPU work.

In a ML pipeline its time is high while the GPU's input wait is near zero, because the fetch overlaps the previous step.

Where I ended up: record both clocks for every phase, pick ONE clock per analysis window (and say which), report never-measured as null instead of 0.0, and only compare runs on a clock both measured.

How do you handle this in your own timing code: sync and eat the stall, or keep the two clocks separate?

What it does: attaches forward/backward hooks across your whole model, tracks stats per-module (mean, std, skew, kurtosis, zero-fraction, KL-to-unit-gaussian, etc.), and gives you a GUI to explore it: a tree view of the model where you can click into any layer and plot its stats over time, plot gradient flow across the network (or grouped by layer type), and log/plot arbitrary tensors like loss or custom metrics.

Uses torch.fx to trace execution order so the plots are laid out in actual model depth order, not just module registration order. Hooks are meant to be attached/detached manually (e.g. every Nth training step) so it doesn't tank your training speed if left on the whole run.

Tested it on a flow-matching U-Net (~10M params) trained on CIFAR-10 for a few epochs — screenshots in the repo. I fired the hooks every 10th iteration and that resulted in 3.5% higher training time.

Still early, would appreciate any feedback!

https://github.com/leonardozh1709/ModelAnalyzer

Sharing a project that might be useful to people who think about model architecture visually: NeuroBranch keeps a graph and its generated PyTorch in sync in both directions. You build the graph, it compiles to real PyTorch through a dialect compiler — but you can also edit the supported PyTorch constructs directly and have those edits parsed back into the graph.

Execution runs on a local Python runtime (atomic_runtime.py) reachable via IPC, with run/rerun/reset and step-by-step tensor inspection. Ports are typed at the IR level, so the graph enforces shape/type compatibility before anything compiles.

Core is framework-agnostic (typed IR, compiler, topology-aware layout) sitting under an Electron/React shell. There's also a reusable-card studio for writing your own nn.Module cards, constrained to explicitly supported torch.nn constructors — no arbitrary code eval.

Repo: https://github.com/sanjayrohith/NeuroBranch (Apache-2.0)

Curious what this community thinks of the two-way sync approach specifically, and where the dialect parser would break on real-world architectures — that's the part most likely to have edge cases right now. Contributions and bug reports welcome.

I have created a simple C++ framework for LibTorch - https://github.com/MartinPerry/LibTorchFramework/tree/master.

Sadly, it cannot currently be compiled since it relies on a proprietary library and the code is not "cleaned" of hard-coded paths, etc.

Is it useful? Probably not :-). A lot of things need to be rewritten that are not part of LibTorch (but are present in PyTorch) - for this, I have used LLMs (it is quite handy for conversion of model structures from PyTorch to C++ with LibTorch).

However, I am sharing it so that someone can reuse parts of the code or be inspired in their own project if they want to use C++.

I'm trying to use OmniVoice Studio to dub non-English videos into English. In the setup however, I run into the following error.

I've tried using the stable and nightly versions of Pytorch for CUDA 12.6, 13.0, and 13.2.

Here is the output from a window command line to verify.

>>> print(f"PyTorch version: {torch.__version__}")
PyTorch version: 2.13.0+cu132

>>> print(f"CUDA Version: {torch.version.cuda}")
CUDA Version: 13.2

>>> print(f"Device Name: {torch.cuda.get_device_name(0)}")
Device Name: NVIDIA GeForce RTX 4070 SUPER

>>> x = torch.rand(5, 3).cuda()

>>> print("GPU Tensor Test Successful:", x.device)
GPU Tensor Test Successful: cuda:0

From what I read, there should be backwards/forwards compatibility (?) as the 4070 Super uses sm_89.

Is the program looking for sm_89 to be explicitly stated in the TORCH_CUDA_ARCH_LIST? If so, how to I fix this?

Or am I missing a simple solution?

Edit: 4070 SUPER not Ti, can't edit the title :(

SOLVED:
There's a variable called OMNIVOICE_FORCE_CUDA that has to be set to 1. Was able to do so by running the following command in powershell:

[Environment]::SetEnvironmentVariable("OMNIVOICE_FORCE_CUDA","1","User")

Now the warning still appears, but the GPU is now used for OmniVoice.

I have been training custom models for a few years now in the finance realm. I barely have any transformer layers and half the time they are custom so flash attention isn't something I need.

With the 9070 xt being $750 ish and the rumor is a 5070 ti super will be like $1400 (seriously nvidia go F#_& yourself) I wonder if for $750 the AMD card would work well for me. I already have a 3060 12gb and a 5060ti 16gb churning out test runs, but I want to add another card. I am nowhere near vram limited. My bottleneck is strictly more compute/bandwidth.

Would I regret getting a 9070 XT? Supposedly support is way better than it used to be. Also I run linux. Windows is garbage.

Hi everyone,

I'm a 3rd-year Electrical and Electronics Engineering student interested in embedded systems. My goal is to become an Embedded AI/Edge AI engineer.

I've already started learning Embedded C (STM32, microcontrollers) and today I'm starting PyTorch. Eventually, I want to train models in PyTorch and deploy them on embedded hardware like STM32 (TinyML) and NVIDIA Jetson.

I'd appreciate advice from people working in this field:

What learning roadmap would you recommend?

Which topics in PyTorch should I focus on for Edge AI?

What projects would make my resume stand out?

Are there any books, courses, or GitHub repositories you wish you'd known about when you started?

What mistakes should I avoid

Take a look at the schedule for PyTorch Conference North America (Oct. 20-21 in San Jose, CA)
View the agenda live now
Submit a poster by July 26th
Register - early bird conference passes are available at a discount through July 31st

Hello Peeps!

Do you guys do a lot of training or fine tuning? Does the loss curve look fine, but the run is slower than it should be, and figuring out why usually means firing up a profiler and staring at a trace for twenty minutes?

This got me curious: what this actually costs, tool by tool. I took one run I knew was input-bound (dataloader starving the GPU) and measured it three ways: torch.profiler, cProfile, and TraceML, a lighter always-on OSS tool I've been contributing to.

For each one I looked at overhead, how much the profiler itself perturbs the GPU utilization it's trying to measure, output size, and how much manual digging it takes to get from the raw output to "the dataloader is the problem."

Short version: torch.profiler and cProfile are precise but heavy and after the fact, closer to a scalpel. Something that just sits there and flags "this step looks off" while training runs is doing a different job, not replacing them.

Numbers and traces are in the post.

Curious how other people usually catch this before it burns your precious compute.

https://medium.com/traceopt/traceml-vs-torch-profiler-vs-cprofile-what-each-one-costs-to-find-the-same-bottleneck-745a57e13ee9?sharedUserId=apendyala