r/programming 15h ago

Bazel's UX is really, really, really bad

https://blog.appliedcomputing.io/p/bazels-ux-is-really-really-really
255 Upvotes

124 comments sorted by

170

u/FlukeHawkins 14h ago

I discussed Bazel with a fairly senior guy from Uber and his advice to me was "don't bother unless you can afford dedicated resources for upkeep"

73

u/comrade_donkey 13h ago

Yes, this! Realistically, for any non-trivial codebase, your deps are not going to "just work" with Bazel.

You're gonna need a small team of people that maintains Bazel itself, build rules, the remote execution engine, cache, and patches for dependencies' auto-generated BUILD files.

40

u/randylush 11h ago

that's been my experience with all of Google's infrastructure tools. They don't actually reduce costs at all, they just separate costs into a bucket for you. Instead of everyone working on build problems, you can move to Bazel and have a dedicated Bazel expert work on Bazel problems. Instead of everyone working on deployment, you can move to Kubernetes and have a Kubernetes expert work on that. The only benefit a lot of these tools provide is being able to better quantify how expensive a given process is.

48

u/SanityInAnarchy 10h ago

It can absolutely reduce costs, it's just an economies-of-scale kinda thing, in two ways:

Google has more than 50,000 software engineers. (Maybe much more, they have like 200k employees, but they don't publish how many are full-time SWEs.) Let's pick that number as a baseline. Now let's say Bazel saves even 1% of your time working on build-system bullshit. All Google has to do for this to be worth it is for their Bazel team to be fewer than 500 full-time engineers maintaining it.

And, they can reduce costs you probably don't have if you aren't as big as them. Google has a monorepo so large that it's probably been more than a decade since it's fit on a single machine! If you have a build that you can run on your laptop in a reasonable amount of time, you do not have a Bazel-shaped problem.

18

u/comrade_donkey 9h ago edited 1h ago

Yes, and Google is an NIH company. While they do mix a light breeze of external deps into their code, 99+% of the monorepo is first-party code, developed at Google, on Blaze, with first-party BUILD files.

In most other repos, third party dependency code dominates over first party. Bazel needs a BUILD file for every dep. There's auto-generators. They suck. You need to write a patch file and patch the auto-generated BUILD file. No, really.

When you update your deps, your patch files predictably stop working. Need to update 12 deps? Be ready to write 12 new patch files. It fucking sucks.

10

u/SanityInAnarchy 9h ago

I've seen at least some generators work okay -- you can add Python pip dependencies easily enough, for example. But the way I've seen this done at places that are less NIH is, you end up with Bazel functioning more as a wrapper around the third-party dep, rather than actually natively generating BUILD files for every import in the library.

Which is... fine, but the more you do that, the more you're kind of losing the point of Bazel in the first place. Especially if you ever have to fork that dependency.

Google, meanwhile, would rather vendor absolutely everything, including the Python interpreter itself, and also wholesale replace the vendor's build system with hand-written Bazel, to the point where your builds are so hermetic that you compile the Python interpreter from source to build your python scripts.

I again think it's not as absurd as it sounds, if you are Google. That work you're describing is easy to write off when those 12 deps are used by literally dozens of other teams that didn't have to think about deps at all. The same logic kinda leads to NIH, though -- you can also amortize the cost of building it yourself, instead of adding that dep.

2

u/inio 2h ago edited 2h ago

to the point where your builds are so hermetic that you compile the Python interpreter from source to build your python scripts.

And thank the Holy Mother of all SWE Gods and Linux Kernel Engineers that there's a global build cache to make that far less insane than it sounds.

1

u/ricky_clarkson 1h ago

Most Google BUILD files are not hand written. I don't know if build_cleaner is open sourced, but it works well internally.

1

u/comrade_donkey 1h ago edited 1h ago

It is not, to my knowledge. It helps in the common cases, for sure. But it's not magic. I've written many a #keep directive. By 'hand written' I mean checked in by people alongside their code. Not generated post-facto.

-7

u/sionescu 7h ago

Quite the contrary, it's an "Invented Here" company, because when those internal tools were written, they were the first of their kind. Nothing similar existed elsewhere.

5

u/randylush 7h ago

NIH means “I’m not gonna use that because it wasn’t invented here.” Meaning, Google doesn’t use stuff that they don’t invent. They are a NIH company by definition. Shouldn’t be confusing.

-6

u/sionescu 5h ago

Dude, NIH refers to the tendency of rejecting outside products when they already exist and preferring to do one's own thing. It does not apply to innovators where no such external products existed. A simple concept that even you might be able to understand.

2

u/axonxorz 4h ago

It does not apply to innovators where no such external products existed.

There are plenty of cross-platform, cross-language build systems out there, Bazel isn't a greenfield concept.

-1

u/sionescu 4h ago edited 4h ago

Bazel isn't a greenfield concept

Yes it is. Before it, there were only various Make-style event-based build systems. Bazel is based on a fully materialized compilation tree, with entirely hermetic build nodes, a purely functional build language, a built-in remote build and cacheing protocol for sharing intermediate artifacts between developers. Nothing like it existed at the time.

→ More replies (0)

0

u/SanityInAnarchy 1h ago

Google has been both, for better and worse. But the point isn't whether Bazel itself was an example, we're talking about the ways in which Bazel discourages adopting third-party libraries. Outside Google, here's what I have to do to add a new third-party Python library:

  • Add a line to a config file
  • Regenerate any lockfiles -- literally one uv command
  • Send the PR
  • Close the autogenerated legal-review ticket with "pre-approved license"

What does that process look for you? How about for C++? What if it's a C++ library that throws exceptions?

And how long does it take you to upgrade a library?

Also, if you're still there, maybe look up go/nojerks again.

1

u/sionescu 1h ago

 we're talking about the ways in which Bazel discourages adopting third-party libraries

No we weren't talking about that, at all.

Google internally has importers that can very well convert the most common build systems (automake, CMake, meson, pypi, Go, etc...). The reality is that below the top few hundred libraries for each language, the quality degrades so much that they're not worth investing on. If a library was worth it, it would be very easy to add, but most often it's not.

-3

u/randylush 7h ago

Right, I’m saying it may save 99% of the developers 1% of their time but it actually requires a team of 1% to keep it running. It’s usually a break-even at best. Once you’ve wrestled with Kubernetes or Bazel like OP’s link, or spent days of company time learning about it, the costs build up. It’s more about separating responsibilities, being able to hire fungible technicians, and following industry standards than actual cost saving.

10

u/SanityInAnarchy 7h ago

Yeah, I think that's where we disagree: I don't think it's a linear 1% needed to run the thing. If Google actually has 500 engineers working full-time just making Bazel work, that's break-even. I bet they can do it with 100 or even 50, even including the people developing Bazel itself as a product.

If your entire company is 100 people, then one full-time Bazel expert is break-even, but you probably need more than that, so it makes no sense.

16

u/sionescu 13h ago

Yep. Bazel requires an entire service around it for remote compilation and caching; maintaining the internal rules and the importers that convert the "outside" build systems to Bazel.

2

u/LoweringPass 10h ago

But what exactly is the alternative if you have a polyglot repo?

4

u/tadfisher 8h ago

Nix works for this, for various definitions of "works"

3

u/RockstarArtisan 5h ago

My advice for nix is: "don't bother with developing under nix unless you can afford dedicated resources for upkeep". For deployments it's fine. I have used uber before. I hate uber.

0

u/mpyne 9h ago

Probably something like kde-builder. A simplistic script to checkout, build, install and keep up-to-date multiple repos isn't tremendously complicated to do after all.

19

u/whitechapel8733 11h ago

I think the false narrative is that Bazel is inherently more complex once you have a sufficient number of services.. but once you’ve built enough bespoke stuff across several languages, etc you realize Bazel is actually a simple solution to a very convoluted problem.

5

u/remy_porter 10h ago

I think there is something to a heterogeneous codebase being a good fit to Bazel. I'm working in ROS2, with Python and C++ code, and while we depend on an upstream ruleset for making ROS work, it does just work- once we got it all set up.

9

u/CosmosGame 11h ago

We have a group in my company doing Bazel and they told me exactly the same! I'm not a huge fan of maven, but at least at this point I understand it.

3

u/rsminsmith 7h ago

Yup, I'm part of the group that maintains our bazel setup. IMO it's overkill unless you have a large number of developers, need to support multiple languages, and/or have specific niche use cases. In our case for instance, one of our security requirements involves building every dependency from source w/patches as needed, which would be a nightmare without something like bazel.

72

u/remy_porter 14h ago

Having recently switched from CMake to Bazel I am worlds happier with Bazel. I don’t love it, but at least it’s not CMake.

I really hate CMake.

46

u/jesseschalken 14h ago

Anything looks good compared to CMake.

22

u/remy_porter 14h ago

I have yet to find a build system that isn’t bad. I’ve tried a bunch. It’s not even about finding the least bad, but about finding the “doesn’t stab me with the workflow I’m currently using it for”.

18

u/Icy-Opposite-7890 13h ago

They all suck. What’s important is everyone uses the same sucky system so I don’t have to learn a different sucky system with every project.

15

u/ozyx7 10h ago

Not that long ago make was the sucky system that everyone used, and things were not great but were at least consistent.

Now we have a multitude of different sucky systems.

14

u/almost_useless 9h ago

Not that long ago make was the sucky system that everyone used

It was never the system that everyone used. Approximately nobody on windows used it, but at least almost everyone on *nix used it.

2

u/Icy-Opposite-7890 7h ago

As sibling pointed out, we needed CMake for cross-platform builds. Doubtless there’ll be something new in the future that we’ll all switch to which will suck in its own unique way.

5

u/skydivingdutch 12h ago

Have you tried Meson? I hear good things, but yet to try it myself

5

u/remy_porter 11h ago

I've done some projects that used Meson. Never used it enough to learn why I hate it, but I'm sure if I used it enough I'd hate it.

7

u/LoweringPass 10h ago

CMake is not the problem, the problem is that C++ developers don't know how to use it and building C++ is inherently complicated

17

u/Ok-Scheme-913 10h ago

I mean, cmake has plenty of problems in and of itself. It's a hack of a hack with disgusting syntax.

1

u/VirginiaMcCaskey 45m ago

The average C++ developer can't tell you the difference between the compiler, the linker, and the loader.

That's not a knock on C++ developers. These are excellent abstractions and the fact two generations of engineers could tell you what they do but not how they work is a testament to how much complexity has been shoveled underneath them by people developing toolchains and build systems.

-6

u/satireplusplus 9h ago

They are all pretty much fine once you have Claude dealing with the bullshit. And when you don't have to deal with autoconfs and automakes m4 bs, its actually quite nice too.

2

u/Techrocket9 4h ago

Yeah, Bazel is a humongously complex tool, but it's exactly the right level of complexity for wrangling c++ build systems.

I'll use Bazel even for small c++ projects, though its value really shines on gigantic enterprise codebases.

1

u/Slsyyy 11h ago

I agree. Nowadays I code in go and toolkit is so great that I would not switched until the scale requires it. On the other hand C++ tooling is so cursed that Bazel is miles ahead of anything else

79

u/gladfelter 14h ago

bazel doesn't scale down to one developer well. It's amazing for tens of thousands of developers thanks to its scalability and deep introspection capabilities. You can find and fix build graph problems at scale.

The previous version of blaze did a lot of things by convention and you couldn't trace them without leaky heuristics in some cases. Bazel is completely introspectable and deterministic.

7

u/larsga 10h ago

Used it at a company that had maybe ~10 developers. It worked just fine, and I don't recognize OP's issues at all. Sure, there was a learning curve to get over, but there is with Maven and Gradle, too.

9

u/GlassLost 5h ago

Saying it's better than maven and gradle is a low bar.

1

u/ZCEyPFOYr0MWyHDQJZO4 2h ago

I liked gradle [more than maven].

0

u/larsga 1h ago

I completely agree. I did like the overall model of Bazel, and particularly the clean separation between build specification and extension language. Gradle IMHO messes up the boundaries there quite badly. The way Bazel never does any more build work than necessary is also very nice.

It does seem like it ought to be possible to make much better build tools, but I don't know of any.

-1

u/repster 14h ago

It actually does scale down pretty well if the one developer is backed by Claude. Speaking from experience

75

u/ChemicalRascal 9h ago

Good grief, if I have to burn the planet just to keep a build chain maintained, maybe the build chain is using the wrong tools.

0

u/idebugthusiexist 9h ago

Most levelheaded comment of the day 😂 🏆

11

u/steveklabnik1 13h ago

Yes, this is one of the things that AI really helps with. I'm more of a buck guy than a bazel guy, but all of this is just way way way easier when you can ask for help, and Claude is very good at figuring out what you need to do.

0

u/cafemachiavelli 6h ago

Seconded, even my local qwen model did okay with a bit of hand holding. 

58

u/TouchyInBeddedEngr 15h ago

Just wait until you try to develop build rules.

19

u/lppedd 14h ago

I don't think Bazel was meant for the general public like Maven or Gradle. It just doesn't feel right. It's probably ok if you have a massive project with dozens of different languages that all need to be built at the same time to output something. But even then, you'll need a couple people dedicated to Bazel only.

19

u/sionescu 14h ago

That's correct. Bazel is meant to be used within a tightly controlled enterprise environment.

11

u/Slsyyy 11h ago

It just doesn't feel right, because each language nowadays has a nice and dominant build system, which is tightly coupled to a given technology

It was not true for C/C++ (IMO it is still bad) nor Java/Python those 20 years ago. I feel that Bazel could be dominant build system, if the current version was released those 20 years ago

13

u/jesseschalken 14h ago

The reason Bazel's Rust builds are so slow is because it doesn't pass -Cincremental={some cache} to rustc to do incremental compilation like Cargo does. A crate that was modified is compiled from scratch, every time.

A colleague of mine has a draft PR up to fix it here, but this is always the pattern with Bazel. Out of the box it does lots of things but none of them especially well.

6

u/steveklabnik1 13h ago

I'm more familiar with buck than bazel, but there have been some rustc bugs that make this not okay to pass. see here: https://rust-lang.zulipchat.com/#narrow/channel/131828-t-compiler/topic/Possible.20Reproducibility.20Bug.20in.20Soundness.20Fix.3F/with/616729102

5

u/jesseschalken 13h ago

If it's not okay to pass, why does Cargo still pass it by default?

Maybe if the result is going to be written to a shared cache, the impact of a miscompile could make it not worth the risk.

But for local iteration its indispensable. Bazel is always going to be compared to Cargo for Rust build perf and Cargo currently has a huge edge here.

13

u/steveklabnik1 12h ago

It's fine for Cargo, because Cargo is not a hermetic build system.

That is, the bug makes systems like buck/bazel not actually do incremental builds, it's not a correctness issue.

5

u/singron 8h ago

In particular, non-reproducibility (e.g. embedding a UUID or timestamp) prevents the "early cutoff" optimization (name from this paper). E.g. if you change a comment and recompile with a reproducible process, the compiler object output will probably be unchanged and you can skip recompiling dependent libraries, linking the executable, and running tests.

It's especially bad if the rmeta files are non-reproducible since that would force all dependent crates to recompile, which would force all their dependent crates to recompile, and so on.

1

u/jesseschalken 6h ago

Huh? Bazels incrementality doesn’t depend on the build being hermetic. If an action always produces different outputs, it just defeats early cutoff and consuming actions always rerun. The same as Cmake/Make. It’s still incremental.

1

u/steveklabnik1 2h ago

The build re-running is exactly the issue. It will cause rebuilds that don't actually need to happen. See your sibling comment.

24

u/UltraPoci 15h ago

Wait, Kubernetes is considered bad?

40

u/Aelig_ 14h ago

It's considered "worse" than Borg but only people who worked at Google actually know if that's true or not.

It's generally agreed that Borg can operate at a much larger scale and I've had former Google devs tell me there are some really nice features that are default in Borg and don't exist in k8s but again, there's no way to actually know.

14

u/sionescu 14h ago edited 14h ago

Yes, it's definitely much worse than Borg. Its not a surprise because, even though on of the main authors of Kubernetes was the tech lead of Borg control plane, none of the authors was an SRE, so they didn't really understand what made Borg good, how we SREs were using it (and the tooling around it), and managed to make it worse.

14

u/CoVegGirl 13h ago

There are three kinds of product on Google Cloud:

  1. Products that are a veneer around systems Google uses internally
  2. Half-assed reimplementations of their own systems
  3. Systems that they’d never use in a million years because they’ve implemented a much, much better system internally

Fun fact: when Google App Engine was very first launched way back in 2011, its internal code name was “Prometheus” in that it was intended to give the plebes access to superior Google technology. It got a well-deserved reputation for being very narcissistic.

7

u/sionescu 12h ago

I do get your humour, but it's not really like that: Cloud Spanner is a separate product from the internal Spanner. It's more conservative, probably 1 year late with features, and it has all the cloud tooling not used internally because Annealing is better. Alloydb is also quite good.

For monitoring, it had a language vaguely similar to Mash but it's been deprecated in favour of PromQL which is awful.

In general I find GCP much nicer to use than AWS or Azure.

8

u/CoVegGirl 9h ago

The point is that Google isn’t great at eating its own dogfood internally. You’re right to point out that that’s not entirely a bad thing, and is often a good thing. I guess I don’t have much experience actually using GCP. In fact, that’s the point.

My perspective having worked on GCP is that it just feels really weird to be working on products that there’s no chance in hell they’d ever consider using themselves.

2

u/sionescu 9h ago

Well, that's because what they have internally is still the best enterprise development system and cluster manager ever built even though, due to the insane levels of integration it has, it's very hard to sell outside.

2

u/jmickeyd 3h ago

The problem is Borg isn't flexible enough to be used in the outside world. For example, Kubernetes networking is a shit show because existing software's networking is a shit show. Without BNS, there is no good way to do internal connections. Too many apps just resolve the DNS once at startup and keep connecting to the same IP assuming it will never change. This led to having a layer 3 proxy running everywhere.

2

u/sionescu 2h ago

 The problem is Borg isn't flexible enough to be used in the outside world.

That inflexibility is what makes it so insanely Useful. For 99% of developers it's effectively a gRPC-only network that's only incidentally layered on top of TCP/IP. There's no firewall to deal with, just ACLs based on Ganpati groups. BNS, GSLB with automatic spillover to adjacent cells. This is insanely useful and if you read Reddit and HN, people are still reinventing it on top of Kubernetes + cloud primitives, but worse.

26

u/comrade_donkey 14h ago

They're different. Kubernetes is the whole stack: Resource definitions, reconciliation loop, container scheduling, etc. The Borg is "only" an extremely powerful task scheduler. Prodspec and Annealing are the config & reconciliation loops, and completely independent systems.

9

u/sionescu 14h ago

Exactly ! Finally, someone who knows about Prodspec and Annealing. They're much better designed than any of the app deployment resources in Kubernetes.

1

u/jmickeyd 3h ago

Honestly I kind of hated the way they shoehorned policy into the enforcer. Trying to figure out why something was pinned could be a mess.

7

u/Ignisami 11h ago

He’s not saying kubernetes is bad, he’s saying kubernetes is a watered-down version of what Google uses internally.

10

u/ozyx7 13h ago

My pet peeves are:

  • It makes all build files read-only, making them more annoying to remove later, or making it more annoying if you want to copy those files elsewhere later and the permissions get copied too.
  • It litters my source tree workspace with bazel-bin, bazel-out, bazel-testlogs, and bazel-WORKSPACE symlinks. Those symlinks "are only for the user’s convenience, as Bazel itself does not use them". WTF? If they're there for my convenience, how about letting me opt-out of some them, giving them sensible default names (is bazel-bin or bazel-out where the final output is?), or organizing them into a common parent directory?

14

u/semajames 11h ago

common --symlink_prefix=.bazel/

Now you'll get a .bazel dir with:

$ ls .bazel

bin WORKSPACE out testlogs

56

u/valarauca14 15h ago

then throwing a lot of marketing money at the open-source version until it takes over the world. In this post, I’m not talking about the most famous example of this

Often this isn't Google's marketing. It is usually

  1. Googlers trying to "career build" by showing off what they've done & market themselves for future jobs.
  2. Xoogler's going, "Finally X got open sourced I've missed this".
  3. A handful of operating thetans going, "Holy shit, they actually solved {{PROBLEM}}".
  4. People who got psyoped into believing the hype.

By the time Google starts pushing marketing money behind the OSS initiative. 50-80% of Camp 1 left Google. Camp 2 (and some of Camp 3) is doing talks about their internal success stories. Camp 4 has moved onto a new technology.

13

u/Aelig_ 14h ago

And sometimes the thing is just really nice. There's nothing that comes close to kubernetes and everything nice is built on top of it now so it will probably be there forever.

22

u/cafemachiavelli 15h ago

My personal reason 8: Cache invalidation. I'm working on MLIR and a full build takes upwards of an hour. So if bazel decides that it doesn't recognize my work environment and would rather start from scratch, that's up to an hour of productive time gone. This happened daily until I managed to set the magic pls-don't-lose-your-shit-if-PATH-changes parameter, but that broke it on the server, so we're back to extended coffee breaks. Now I've just hardcoded the whole config and I pray that it works. 

13

u/DeepReputation 10h ago

Your build isn’t hermetic. It shouldn’t depend on your work environment, that’s exactly the point.

0

u/max123246 9h ago

Good luck restructuring all of LLVM and MLIR to make it hermetic

6

u/pimp-bangin 14h ago

I think the flag you're thinking of is --incompatible_strict_action_env which fwiw has been turned on by default in newer bazel versions. No clue why the "incompatible" naming or why it was ever off by default, though.

5

u/dacian88 12h ago

incompatible_ flags make breaking changes to the build, it’s an easy way to know as a user that turning something on has a compatibility issue. The manning usually drops after the change is defaulted in after a major version or 2

2

u/Serious-Regular 8h ago

this is why i hate working with G people on MLIR - exactly the same issue: constantly rebuilding because something in env changing.

magic pls-don't-lose-your-shit-if-PATH-changes parameter

what is this param?

2

u/cafemachiavelli 6h ago

experimental_strict_action_env in older bazel versions and as the erudite /u/pimp-bangin mentioned, it's on by default now, so this is partially on our lab for shipping an old version.

My larger gripe would be that setting action env to PATH and/or LD_LIBRARY_PATH ostensibly fixes toolchain discovery issues (and bazel documentation suggests this option), but can cause the cache dump I mentioned. I ended up writing a config script that sets the toolchain directly, once, to have a hermetic build system while keeping the repo somewhat portable.

I still like bazel overall, fwiw, it's still less painful than cmake was. 

24

u/fishheaddz 15h ago

I can't upvote this enough. The only way bazel will work in your organization is by having some kind of true convert who will unfuck bazel full time for everyone. The unitiated will react with utter horror at the unusability of the system and you need this kind of knight to slay the demons. Google even canceled trying to move Android to Bazel after years and millions spent making it work. They can't even get some of their own employees and Android partners to use it.

9

u/IpsumRS 14h ago

You've just described our bazel journey to the letter

4

u/max123246 9h ago

Sounds like CMake. I sometimes think no good build tools end up built because people are exhausted from dealing with the bad ones.

3

u/chengiz 6h ago

Yep and that's why we use make. It's not the easiest thing to set up or use but it's the easiest to figure out what's wrong and how to fix, because it's inherently simple. In the long run it wins.

3

u/Massive_Season7075 9h ago

Are there any well-known apps that use it? And why would someone choose to use if upkeep is costly?

4

u/brianly 6h ago

Google, uber, dropbox, Pinterest, Adobe

5

u/-Knul- 8h ago

Apparently I live under a rock because I never heard of Bazel.

4

u/becauseSonance 7h ago

Can I move in? It sounds lovely

10

u/zoddrick 14h ago

Bazel like other tools built by mega tech corps aren't intended for 99.999% of the engineering orgs on this planet.

3

u/_pupil_ 13h ago

Doing Facebook things without Facebook scale, Facebook demands, and Facebook money is a viable definition of insanity.

3

u/zoddrick 13h ago

Yet here we are 😆

I spend most of my days explaining why we shouldn't use wiz bang thing from goog or meta even though we're a fortune 500 company. But then I also spend my days explaining why we should do other things that just make sense.

You can't explain that....

5

u/vahokif 14h ago

It's really designed for Google with its insane build infrastructure and no one else.

2

u/jonte 10h ago

I've never gotten along with bazel or buck. But the build system used in Chromium, GN, is fantastic. Same-ish syntax, simpler tool.

2

u/Sea-Mistake6086 9h ago

Maybe try the RIGX build system https://github.com/unofficialtools/rigx (TOML configuration, 100% hermetic, Nix powered)

2

u/engineered_academic 5h ago

If you try and use Bazel and Kubernetes to solve a problem, now you have 3 problems

0

u/nickguletskii200 13h ago edited 13h ago

Reason 1: You have the wrong bazel installed

No, you just don't have a .bazelversion file and aren't using bazelisk for some reason.

When I first started working with bazel, I was on a Mac, so I did the natural and normal thing that you do on MacOS: brew install bazel.

Not natural at all. Homebrew is a weird MacOS-ism and I am surprised it's still alive at this point at all.

The official guide recommends the use of bazelisk: https://bazel.build/install/os-x

Anyways, not to spoil the punchline, but apparently the actual tool I needed to install is called bazelisk. Now to be fully transparent: if you read the Bazel docs4, this is their recommended approach, but I’m placing this complaint at the top of my post because it is emblematic of Bazel’s apparent disdain for the people who are actually using the software.

No, if you use an amateur package manager it's not the developer's problem.

If Bazelisk is the tool that developers are supposed to use, why is there a separate homebrew target for bazel?

Fair point. Why does brew even exist still?

Reason 2: Why are there all these BUILD files everywhere?

Now, naturally, any time your build changes (by including a new dependency or source file), you have to update the BUILD file, and thankfully, bazel doesn’t force you to do this by hand. Instead, it provides a tool called gazelle5 to update your BUILD files.

From what I've seen Gazelle is just a way to translate golang's weird import model. At least, I've only seen gazelle used with golang. No need for any of such hacks with rules_rust or other languages I've used.

Reason 4: How do I even build something successfully?

I have no idea what any of that means, except that it’s clearly some dependency that it’s trying to download. Bazel has to know where it’s trying to download from, and it has to know why it’s failing, but it doesn’t actually tell you any of that information, it just gives you an unrelated error message in arcane syntax and leaves you to flounder.

It's literally in the error message above:

curl: (22) The requested URL returned error: 401 WARNING: Download from https://some-url.com/whatever failed: class java.io.FileNotFoundException GET returned 404 Not Found ERROR: no such package '@@some_stuff++whatever+blah_blah_blah_linux_amd64//': oauth failed:

See the WARNING: Download from https://some-url.com/whatever failed: class java.io.FileNotFoundException GET returned 404 Not Found?

But OK, clearly it’s trying to download “something” from “somewhere” and it’s either getting a 401 Unauthorized response OR a 404 Not Found response, and it’s not immediately clear which, but more to the point: this error was indicating that I hadn’t logged in to a private docker registry that was NOT hosted at some-url.com. In fact, nowhere in the entire error message does it say anything about Docker.

Why would it complain about Docker when it's not using Docker? It should be clear from the URL that it's downloading something from an OCI repository.

Also, what on earth is this nonsense? @@some_stuff++whatever+blah_blah_blah_linux_amd64//

You have to learn something new, boohoo. Once you actually learn what this means, come back and tell us how it could be done differently. See https://bazel.build/concepts/labels

Anyways, I logged into the private registry and tried to build again, and was greeted with the following error:

ERROR: no such package '@@+http_io+whatever_v5//file` java.io.IOException: Error downloading [...] GET returned 404 Not Found ERROR: /path/to/BUILD:67:42 //thing/im/building depends on @@+http_io+whatever_v5//file in repository @@+http_io+whatever_v5 which failed to fetch. no such package

OK again so clearly again it’s depending on some package that… doesn’t exist? This makes no sense. Clearly it exists for other people, or everyone would be complaining that everything is broken. Why doesn’t it exist for me?

Finally I got annoyed enough at the CI delays that I decided to revisit, and discovered (after asking some other folks for help) that Bazel was failing because I hadn’t logged into GitHub with the gh CLI app! WTF??? I thought Bazel was supposed to be hermetic? I mean, OK, look, I’m not naïve, sometimes there’s code in a private GitHub repo, and clearly you’re going to have to log in to access it. But again, Bazel doesn’t tell you what code it’s trying to download or why it failed (or even better, say “hey I need some creds can you please provide them”). It just spits out a page of errors in an arcane syntax that refers to packages and modules that you’ve never even heard of before, and expects you to figure it out.

  1. It's pretty clear that it couldn't find that file at that URL. If everyone else isn't complaining, this might be because they have it cached and someone yanked this file from whatever URL you are downloading things from. Nothing to do with Bazel, and this is actually a major reason to use Bazel in the first place: the Bazel external dependencies cache and downloader.
  2. All serious platforms return 404 for resources you have no access to. This isn't a Bazel thing, this is just life working with private registries.

My next complaint is about Bazel’s respect for your (as a developer’s) time. The compile times are through the roof! And I say that as a Rust dev, the language which has as its primary complaint “things take too long to compile”. In some ways I don’t think there’s a lot you can do about this; if you’re trying to make an entirely hermetic build with bit-for-bit reproducibility, that’s going to take more time than “just” running go build. But still. I’ve got stuff to do, I can’t wait 10 minutes for you to finish building my app!

I literally started using Bazel to cut down Rust and OCI compile times. 10 minutes to build an app? If it takes less time than that to build cargo or whatever the golang devs use, something is wrong on your end. Maybe it's sandboxing on MacOS X? Or, perhaps, your target definitions accidentally include tons of data (your IDE's target directory, perhaps)?

Reason 7: Ouch, you punched me in the disk space!

To try to alleviate some of the compile-time issues and need to download every single dependency on the planet for every build, Bazel maintains an absolutely immense local cache on your machine. One of my colleagues was bragging about their 1TB local Bazel cache. WTF??? Look I know we all grew up in an era where disk space was cheap, but it sure isn’t anymore9. And even if it was, seriously? A terabyte of data just to compile some nonsense? I get mad when Cargo doesn’t clean up after itself. I get mad because Docker just likes to vomit all its stuff all over your hard drive indefinitely10. A freaking terabyte???? My god, what have we done.

Pretty normal when you are working with large projects.

Reasons 8, 9, and 10: To be continued.

I am just on the very beginning of my Bazel journey, and have captured in this blog post all (or at least most) of my complaints with Bazel’s user experience. But I’m a fairly new Bazel user, and I’m sure that given my experience thus far, there are going to be many more reasons in the future that I run into with this software, so I’ve left space in this blog post to account for those as well.

Wait until you have to figure out how to define your own toolchains and work with transitions.

And look, this is the Internet. I’m sure some of you will come along and say “Well if you just understood the system” or “This question is answered in the docs” or some other variant of “Git Gud, Noob”. My rejoinder to that is: I am a pretty smart person in general. I am able to reason about some pretty arcane shit. I like to work on complicated, hard-to-understand systems[11]. And I’m sure that, given enough time, I could “git gud” at bazel.

[11] Case in point: I run Kubernetes, at home, for fun! It is well-established at this point that I am a sucker for pain.

  1. Not setting a high bar. Kubernetes is neither complicated nor hard-to-understand on its own, though it is suffering from a lack of good introductory material (just like Bazel).
  2. You have to spend time to understand something and why something is built this way.

But here’s the thing: I don’t want to. All I want is to build some source code and produce a binary. I don’t want to have to learn an entirely new language, parse through pages and pages of errors that don’t actually even tell you what the problem is, spend hundreds of dollars on a new disk drive, and waste hours of my life waiting for something to build that used to “just work”.

Well you should quit software development then. All build systems have their own DSLs. When you are building complex software, you'll need disk space for its dependencies, build artifacts, sysroots and scratch files.

But I am extremely unconvinced, unless your name starts with a G and rhymes with Froogal, that you have any of the problems that Bazel solves bad enough to subject every single one of your developers at your organization to a build system that treats its users with such disdain13.

Small projects may not need it, but every time a project grows something like Bazel becomes necessary. The alternative is to reinvent Bazel using [your favorite duct-tape language], badly.

9

u/jesseschalken 13h ago

The alternative is to reinvent Bazel using [your favorite duct-tape language], badly.

The alternative is to reinvent Bazel using [your favorite duct-tape language], but much better.

Bazel will not be the last word in build systems. It has far too many design flaws and backwards compatibility constraints. The "phases" are an arbitrary holdover from gconfig. The whole repository rule system is basically a second poorly designed build system embedded inside the loading phase.

The world needs a cleaner, simpler, more powerful build system, with a coherent design that isn't just the artifact of some Googler deciding to generate Makefiles with Python 20 years ago.

1

u/YourLizardOverlord 8h ago

I'm using yocto. Does bazel do stuff that yocto doesn't?

0

u/nickguletskii200 13h ago

You are free to do so. Unfortunately, there's nothing even nearly as good as Bazel publicly available at the moment. I would be very excited if you were manage to build something better.

5

u/zokier 10h ago

buck2 is better in almost all aspect except ecosystem?

3

u/nickguletskii200 8h ago

That is one large "except". I would really love to try out Buck2 but then I would have to re-implement half of what the Bazel ecosystem provides out of the box.

2

u/meetcassette 1h ago

I'd love to see this gap closed, did you get hit by particular things like particular prelude gaps or local sandboxing or third-party dep integration? Or was it more of a general feeling.

1

u/Macrobian 3h ago

no sandboxing

6

u/drmorr0 11h ago

So your response to

> I’m sure some of you will come along and say [...] “Git Gud, Noob”.

is to tell me to "Git Gud, Noob"? Lmao, this really is an internet.

5

u/nickguletskii200 7h ago

Nothing in my post says "git gud", though a bit of "RTFM" would be appropriate (even though Bazel's documentation sucks).

My response to you is to stop thinking that you can "just" get rid of all this complexity and make the UX better, because:

  1. Eventually you will encounter the issues that tools like Bazel solve and then you will be glad they are designed this way.
  2. Many of your complaints are not actually about Bazel.
  3. Knowing Kubernetes or any other system you consider "complicated" doesn't make you immediately qualified to judge whether something should be easy for you to get started with.

2

u/brianly 6h ago

Fair responses that don’t deserve the downvotes

1

u/PrimozDelux 5m ago

Bazel has UX so bad you'd think it was a design goal. It really sucked the joy out of work for me for 2 years until AI became a thing and now I no longer have to suffer it, it's Claude's problem now.

-1

u/keithstellyes 9h ago

Funny how competitive a bespoke Python script that just generate a makefile is lol.

-2

u/TCIHL 13h ago

Your first mistake was using homebrew. Just use macports. It works great, and was designed by an Apple employee.

Also, I'm not really sure what the product is trying to solve that regular ol' Make can't do. If you're looking for a byte-for-byte artifact result couldn't you just add a md5 hash check at the end of the process?

3

u/12destroyer21 9h ago

Ideally you should only need to have bazel and maybe git installed on your machine. All other tools should be pulled by bazel to ensure hermeticity 

-5

u/Farados55 14h ago

Ok but google will tell you it necessary