From: Linus Torvalds <torvalds@osdl.org>
Newsgroups: fa.linux.kernel
Subject: Re: [patch 00/2] improve .text size on gcc 4.0 and newer compilers
Date: Thu, 29 Dec 2005 17:42:32 UTC
Message-ID: <fa.hbk15l5.nn82g1@ifi.uio.no>
Original-Message-ID: <Pine.LNX.4.64.0512290923420.14098@g5.osdl.org>

On Thu, 29 Dec 2005, Ingo Molnar wrote:
>
> * Andrew Morton <akpm@osdl.org> wrote:
> >
> > When it comes to inlining I just don't trust gcc as far as I can spit
> > it.  We're putting the kernel at the mercy of future random brainfarts
> > and bugs from the gcc guys.  It would be better and safer IMO to
> > continue to force `inline' to have strict and sane semamtics, and to
> > simply be vigilant about our use of it.
>
> i think there's quite an attitude here - we are at the mercy of "gcc
> brainfarts" anyway, and users are at the mercy of "kernel brainfarts"
> just as much.

There's a huge difference here. The gcc people very much have a "Oh, we
changed old documented behaviour - live with it" attitude, together with
"That was a gcc extension, not part of the C language, so when we change
how gcc behaves, it's _your_ problem" approach.

At least they used to.

So yes, there's a huge attitude difference. The gcc people have a BAD
attitude. When the meaning of "inline" changed (from a "inline this" to
"hey, it's a hint"), the gcc people never EVER said "sorry". They
effectively said "screw you".

I know this is why I don't trust gcc wrt inlining. It's not so much about
any technical issues, as about the fact that the kernel tends to be a lot
heavier user of gcc features than most programs, and has correctness
issues with them, AND THE GCC PEOPLE SIMPLY DON'T CARE.

Comparing it to the kernel is ludicrous. We care about user-space
interfaces to an insane degree. We go to extreme lengths to maintain even
badly designed or unintentional interfaces. Breaking user programs simply
isn't acceptable. We're _not_ like the gcc developers. We know that
people use old binaries for years and years, and that making a new
release doesn't mean that you can just throw that out. You can trust us.

Maybe gcc development has changed. Maybe it hasn't.

THAT is what makes me worry. I don't know if this is why Andrew doesn't
trust inlining, but I suspect it has similar roots. Not trusting it
because we haven't been able to trust the people behind it. No heads-up,
no warnings, no discussions. Just a "screw you, things changed, your
usage doesn't matter, and we're not even interested in listening to you
or telling you why things changed".

There have been situations where documented gcc semantics changed, and
instead of saying "sorry", the gcc people changed the documentation. What
the hell is the point of documented semantics if you can't depend on them
anyway?

One thing we could do: I think modern gcc's at least have an option to
warn when they don't inline something. It might make sense to just enable
that warning, and see _which_ functions -Os and -funit-at-a-time say are
too large to be inlined.

Maybe the right thing to do is to just heed that warning, and remove such
functions from header files and make them no-inline? That way we get the
size fixes _regardless_ of any compiler options.

				Linus

From: Linus Torvalds <torvalds@osdl.org>
Newsgroups: fa.linux.kernel
Subject: Re: [patch 00/2] improve .text size on gcc 4.0 and newer compilers
Date: Thu, 29 Dec 2005 20:50:52 UTC
Message-ID: <fa.g0pt5rc.ig04qm@ifi.uio.no>
Original-Message-ID: <Pine.LNX.4.64.0512291240490.3298@g5.osdl.org>

On Thu, 29 Dec 2005, Dave Jones wrote:

> Seriously, we break things _every_ release. Sometimes in tiny
> 'doesn't really matter' ways, sometimes in "fuck, my system no
> longer works" ways, but the days where we I didn't have to tell
> our userspace packagers to rev a half dozen or so packages up to the
> latest upstream revisions when I've pushed a rebased kernel are
> a distant memory.

Umm.. Complain more. I upgrade kernels a lot more often than I upgrade
distros, and things don't break. They're not allowed to break, because I
refuse to upgrade my user programs just because I do kernel development.
But I'd only notice a small part of user space, so if people don't
complain, they break not because we don't care, but because we didn't even
know.

So if you have a user program that breaks, _complain_. It's really not
supposed to happen outside of perhaps kernel module loaders etc things
that get really really chummy with kernel internals (and even that was
fixed: the modern way of loading modules isn't that chummy any more, so
hopefully we'll not need to break even module loaders again).

If we change some /proc file thing, breakage is often totally
unintentional, and complaining is the right thing - people might not even
have realized it broke.

At least _I_ take breakage reports seriously. If there are maintainers
that don't, complain to them. I'll back you up. Breaking user space simply
isn't acceptable without years of preparation and warning.

			Linus

From: Linus Torvalds <torvalds@osdl.org>
Newsgroups: fa.linux.kernel
Subject: Re: [patch 00/2] improve .text size on gcc 4.0 and newer compilers
Date: Thu, 29 Dec 2005 21:26:39 UTC
Message-ID: <fa.g19p5be.i0s5ak@ifi.uio.no>
Original-Message-ID: <Pine.LNX.4.64.0512291322560.3298@g5.osdl.org>

On Thu, 29 Dec 2005, Linus Torvalds wrote:

> At least _I_ take breakage reports seriously. If there are maintainers
> that don't, complain to them. I'll back you up. Breaking user space simply
> isn't acceptable without years of preparation and warning.

Btw, sometimes we knowingly change semantics that we believe that nobody
would ever be able to care about. Then we literally _depend_ on people
complaining about breakage in case we were wrong, and if you guys don't,
and just curse, and upgrade programs, we actually miss out on real
information.

And yes, occasionally we don't have much choice, and things break. It
should be extremely rare, though. Much more commonly it would be a bug or
an unintentional change that somebody didn't even realized changed
semantics subtly.

		Linus

From: Linus Torvalds <torvalds@osdl.org>
Newsgroups: fa.linux.kernel
Subject: Re: userspace breakage
Date: Thu, 29 Dec 2005 22:57:34 UTC
Message-ID: <fa.g0pn63d.igm4in@ifi.uio.no>
Original-Message-ID: <Pine.LNX.4.64.0512291451440.3298@g5.osdl.org>

On Thu, 29 Dec 2005, Dave Jones wrote:
>
> At some point in time it became defacto that certain things like udev, hotplug,
> alsa-lib, wireless-tools and a bunch of others have to have kept in lockstep
> with the kernel, and if it breaks, it's your fault for not upgrading
> your userspace.

Hmm.. Time for some re-indoctrination?

We really shouldn't allow that. I know who to blame for udev, who else
should I complain to?

> Just a few years ago, if someone suggested breaking a userspace
> app in a kernel upgrade, they'd be crucified on linux-kernel, now
> it's 'the norm').

That really isn't acceptable. Breaking user space - even things that are
"close" to the kernel like udev scripts and alsa-lib, really is NOT a good
idea.

We're much better off wasting a bit of time on backwards compatibility,
than wasting a lot of user time and irritation (and indirectly, developer
time) on linkages to packages outside the kernel.

If you cannot upgrade a kernel without upgrading some user package, that
should be considered a real bug and a regression.

There are real technical reasons for not allowing those kinds of version
linkages: it makes it MUCH harder to blame the right thing when things go
wrong.

Now, I'm not saying that we can always support everything that goes on in
user space forever, but dammit, we can try damn hard.

(Somehow I'm not surprised about alsa. I think the whole alsa release
process has always sucked. Dang).

			Linus

From: Linus Torvalds <torvalds@osdl.org>
Newsgroups: fa.linux.kernel
Subject: Re: userspace breakage
Date: Thu, 29 Dec 2005 23:56:33 UTC
Message-ID: <fa.g09j5rg.h0u4qi@ifi.uio.no>
Original-Message-ID: <Pine.LNX.4.64.0512291544310.3298@g5.osdl.org>

On Fri, 30 Dec 2005, Adrian Bunk wrote:
> >
> > Now, I'm not saying that we can always support everything that goes on in
> > user space forever, but dammit, we can try damn hard.
>
> Was it a mistake to drop support for ipfwadm and ipchains?
> Was it a mistake to drop support for devsfs?
> Will it be a mistake to drop support for gcc < 3.2?

Those things at least were brewing for _years_. People had lots of
heads-up warning.

> Will it be a mistake to remove the obsolete raw driver?
> Will it be a mistake to drop the Video4Linux API 1 ioctls?
> Will it be a mistake to drop support for pcmcia-cs?

And again, this is something that we've been warning about. We have.

I'm not talking about never obsoleting bad interfaces at all. I'm talking
about the unnecessary breakage that comes from changes that simply aren't
needed, and that isn't given proper heads-up for.

We used to have a fairly clear point where we could break things, when we
had major kernel releases (ie 2.4 -> 2.6 broke the module loader. It was
documented, and it was unavoidable).

> The fundamental problem is that the current development model
> contains no well-defined points where breakages of the kernel-related
> userspace were allowed and expected by users.

The basic rule should be "never". For example, we now have three different
generations of the "stat()" system call, and yes, we wrote the code to
maintain all three interfaces. Breaking an old one for a new better one
simply wasn't an option.

Now, the more specialized the usage is, the less strict the "never"
becomes. But if something becomes a pain for distribution managers (and
from Dave, it sounds like we've hit that way too often), that definitely
means that we've broken too many things.

In short: I don't think anybody can complain about devfs-like things.
We've kept it up for a _long_ time, and there was tons of help for the
migration. But clearly DaveJ is unhappy, and that implies that we're not
doing as well as we should.

(Which is not to say that we should necessarily bend over backwards to
make sure that DaveJ is _never_ unhappy. We should just try harder).

			Linus

From: Linus Torvalds <torvalds@osdl.org>
Newsgroups: fa.linux.kernel
Subject: Re: userspace breakage
Date: Thu, 29 Dec 2005 23:43:26 UTC
Message-ID: <fa.g0pl5rd.igo4qn@ifi.uio.no>
Original-Message-ID: <Pine.LNX.4.64.0512291541420.3298@g5.osdl.org>

On Thu, 29 Dec 2005, Jeff V. Merkey wrote:
>
> The breakage issue is ridiculous, assinine, and unnecessary. I have been
> porting dsfs to the various releases over the past month, and the
> breakage of user space, usb, nfs, memory management, is beyond absurd.

We're not talking about internal kernel stuff. Internal kernel stuff
_does_ get changed, and we dont' care about breakage of out-of-kernel
stuff. That's fundamental.

		Linus

From: Linus Torvalds <torvalds@osdl.org>
Newsgroups: fa.linux.kernel
Subject: Re: userspace breakage
Date: Fri, 30 Dec 2005 00:11:09 UTC
Message-ID: <fa.g19j4rf.i0q5ql@ifi.uio.no>
Original-Message-ID: <Pine.LNX.4.64.0512291603500.3298@g5.osdl.org>

On Thu, 29 Dec 2005, Jeff V. Merkey wrote:
>
> Linus Torvalds wrote:
> >
> > We're not talking about internal kernel stuff. Internal kernel stuff _does_
> > get changed, and we dont' care about breakage of out-of-kernel stuff. That's
> > fundamental.
>
> Start caring. People spend lots of money supporting you, and what you are
> doing. How about taking some responsibility for that [...]

Cry me a river, Jeff.

The kernel is GPL'd. That's my responsibility. Source code. Stuff that
comes to me as patches. That's my job, and that's what I get paid for. In
fact, my contract says that I _cannot_ work on anything that isn't open
source.

Stuff outside the kernel is almost always either (a) experimental stuff
that just isn't ready to be merged or (b) tries to avoid the GPL.

Neither is worth a _second_ of anybodys time trying to support, and when
you say "people spend lots of money supporting you", you're lying through
your teeth. The GPL-avoiding kind of people don't spend a dime supporting
me, they spend their money actively trying to debase and destroy what I
and thousands of others have been working our butts off for.

So don't try to make it sound like something it isn't. We support outside
projects a hell of a lot better than we'd need to, and I can tell you that
it's mostly _me_ who does that. Most of the core kernel developers argue
that I should support less of it - and yes, they are backed up by lawyers
at their (sometimes quite big) companies.

So be honest now. Are those projects you care about going to be GPL'd and
actively pushed back into the standard kernel?

And if they aren't, SHUT THE HELL UP, because they are total freeloaders,
and claiming that they "support" me is total crap.

			Linus

From: Linus Torvalds <torvalds@osdl.org>
Newsgroups: fa.linux.kernel
Subject: Re: userspace breakage
Date: Fri, 30 Dec 2005 00:32:58 UTC
Message-ID: <fa.g19p5bk.i0s5au@ifi.uio.no>
Original-Message-ID: <Pine.LNX.4.64.0512291628530.3298@g5.osdl.org>

On Thu, 29 Dec 2005, Jeff V. Merkey wrote:
>
> The fact that Oracle and IBM support apps on Linux are Freeloading? Baloney!

Jeff, give it a rest.

Oracle and IBM haven't been complaining, have they? Oracle mostly does
user-space stuff (that doesn't change), and has been pretty good about
their Oracle-fs too - they're even actively discussing "git" issues on the
git mailing list, and asking for help, and just generally being good
members of the community.

And IBM engineers are part of the people who change internal kernel
interfaces in order to make it work better for them.

Pretty much the ONLY people who ever complain about those internal kernel
interfaces changing are the free-loaders. It's hard for them, because they
don't want to play according to the rules. Tough. Watch me not care:

   [ Linus sits in his chair, patently not caring ]

See?

		Linus

From: Linus Torvalds <torvalds@osdl.org>
Newsgroups: fa.linux.kernel
Subject: Re: userspace breakage
Date: Fri, 30 Dec 2005 00:49:58 UTC
Message-ID: <fa.fvq35rf.hga4ql@ifi.uio.no>
Original-Message-ID: <Pine.LNX.4.64.0512291643280.3298@g5.osdl.org>

On Thu, 29 Dec 2005, Ryan Anderson wrote:
>
> This, for what it's worth, is the same breakage that Dave seemed to be
> most frustrated with during his OLS keynote, regarding ALSA versions,
> and a few other things that caused breakage and the user space failed to
> work correctly when the kernel was reverted.  (I hope I'm not putting
> words in your mouth, Dave).

I agree: the worst part of version dependency is that it's really hard in
general to just move one of the components backwards or forwards.
Something you want to do when breakage occurs, or just because you need to
figure out some _other_ problem (like doing a kernel bug bisection).

Which is why pretty much _every_ component needs to be backwards
compatible at least to some degree. Otherwise they'd need to be bundled
and developed together as one thing.

IOW, the same way it's wrong for the kernel to need new binaries, it's
wrong for binaries to need a new kernel. It's one reason why we seldom add
new system calls: they aren't all that useful in any kind of short
timeframe, because even programs that would _like_ to use them usually
can't do so for a long time (until they don't have to worry about people
running old kernels any more).

(Some system calls are easier to add than others - if you can easily
emulate the new system call semantics with just a slight performance hit,
you can just have a simple wrapper with a fallback. That doesn't always
work well - some things are just very hard to emulate efficiently).

		Linus