From: Linus Torvalds <torvalds@osdl.org>
Newsgroups: fa.linux.kernel
Subject: Re: git pull on Linux/ACPI release tree
Date: Tue, 10 Jan 2006 19:30:42 UTC
Message-ID: <fa.fv9n3b6.j067qs@ifi.uio.no>
Original-Message-ID: <Pine.LNX.4.64.0601101111110.4939@g5.osdl.org>

On Tue, 10 Jan 2006, Linus Torvalds wrote:
>
> Now, the git history is _not_ really a two-dimensional surface, so it's
> just an analogy, not an exact identity. But from a visualization
> standpoint, it's a good way to think of each "git bisect" as adding a
> _line_ on the surface rather than a point on a linear line.

Actually, the way I think of it is akin to the "light cones" in physics. A
point in space-time doesn't define a fully ordered "before and after": but
it _does_ describe a "light cone" which tells you what is reachable from
that point, and what that point reaches. Within those cones, that
particular point ("commit") has a strict ordering.

And exactly as in physics, in git there's a lot of space that is _not_
ordered by that commit. And the way to bisect is basically to find the
right points in "git space" to create the right "light cone" that you
find the point where the git space that is reachable from that commit has
the same volume as the git space that isn't reachable.

And maybe that makes more sense to you (if you're into physics), or maybe
it makes less sense to you.

Now, since we always search the "git space" in the cone that is defined by
"reachable from the bad commit, but not reachable from any good commit",
the way we handle "bad" and "good" is actually not a mirror-image. If we
fine a new _bad_ commit, we know that it was reachable from the old bad
commit, and thus the old bad commit is now uninteresting: the new bad
commit forms a "past light cone" that is a strict subset of the old one,
so we can totally discard the old bad commit from any future
consideration. It doesn't tell us anything new.

In contrast, if we find a new _good_ commit, the "past light cone" (aka
"set of commits reachable from it") is -not- necessarily a proper superset
of the previous set of good commits, so when we find a good commit, we
still need to carry the _other_ good commits around, and the "known good"
universe is the _union_ of all the "good commit past lightcones".

Then the "unknown space" is the set difference of the "past lightcone of
the bad commit" and of this "union of past lightcones of good commits".
It's the space that is reachable from the known-bad commit, but not
reachable from any known-good commit.

So this means that when doing bisection, what we want to do is find the
point in git space that has _new_ "reachability" within that unknown space
that is as close to half that volume as space as possible. And that's
exactly what "git-rev-list --bisect" calculates.

So every time, we try to either move the "known bad" light-cone down in
time in the unknown space, _or_ we add a new "known good" light-cone. In
either case, the "unknown git space" keeps shrinking by half each time.

("by half" is not exact, because git space is not only quanticized, it
also has a rather strange "distance function". In other words, we're
talking about a rather strange space. The good news is that the space is
small enough that we can just enumerate every quantum and simply
calculate the volume it defines in that space. IOW, we do a very
brute-force thing, and it works fine).

			Linus

From: Linus Torvalds <torvalds@osdl.org>
Newsgroups: fa.linux.kernel
Subject: Re: rc6 keeps hanging and blanking displays where rc4-mm1 works
Date: Fri, 12 Aug 2005 16:54:50 UTC
Message-ID: <fa.g0ad5b8.m0o5qk@ifi.uio.no>
Original-Message-ID: <Pine.LNX.4.58.0508120937140.3295@g5.osdl.org>

On Fri, 12 Aug 2005, Helge Hafting wrote:
>
> > at the moment. The setup is fine with 2.6.13-rc4-mm1 x86-64, no
> > problems there.
>
> The problem still exists in 2.6.13-rc6.  Usually, all I get is a
> suddenly black display, solveable by resizing.

Is there any chance you could try bisecting the problem? Either just
binary-searching the patches or by using the git bisect helper scripts?

Obviously the git approach needs a "good" kernel in git, but if
2.6.13-rc4-mm1 is ok, then I assume that 2.6.13-rc4 is ok too? That's a
fair number of changes:

	 git-rev-list v2.6.13-rc4..v2.6.13-rc6 | wc
	    340     340   13940

but if you can tighten it up a bit (you already had trouble at rc5, I
think), it shouldn't require testing more than a few kernels.

Git has had bisection support for a while, but the helper scripts to use
it sanely are fairly new, so I think you'd need the git-0.99.4 release for
those. But then you'd just do

	git bisect start
	git bisect bad v2.6.13-rc5
	git bisect good v2.6.13-rc4

and start bisecting (that will check out a mid-way point automatically,
you build it, and then do "git bisect bad" or "git bisect good" depending
on whether the result is bad or good - it will continue to try to find
half-way points until it has found the point that turns from good to
bad..)

		Linus

From: Linus Torvalds <torvalds@osdl.org>
Newsgroups: fa.linux.kernel
Subject: Re: Simple script that locks up my box with recent kernels
Date: Sat, 07 Oct 2006 21:27:08 UTC
Message-ID: <fa./oMGTFml4fnJMGJZgMEDjT9Ip0Q@ifi.uio.no>

On Sat, 7 Oct 2006, Jesper Juhl wrote:
>
> > Can I bother you to just bisect it?
>
> Sure, but it will take a little while since building + booting +
> starting the test + waiting for the lockup takes a fair bit of time
> for each kernel

Sure. That said, we've tried to narrow down things that took hours or days
(under real loads, not some nice test-script) to reproduce, and while it
doesn't always work, the real problem tends to be if the problem case
isn't really reproducible. It sounds like yours is pretty clear-cut, and
that will make things much easier.

> and also due to the fact that my git skills are pretty
> limited, but I'll figure it out (need to improve those git skills
> anyway) :-)

"git bisect" in particular isn't that hard to use, and it will really do
a lot of heavy lifting for you.

Although since it will just select a random commit (well, it's not
"random": it's strictly as half-way as it can possibly be, but it's
automated without any regard for anything else), you can sometimes hit a
situation where git will ask you to test a kernel that simply doesn't work
at all, and you can't even test whether it reproduces your particular bug
or not.

For example, "git bisect" might pick a kernel that just doesn't compile,
because of some stupid bug that was fixed almost immediately afterwards.
In those cases, the total automation of "git bisect" ends up being
something that has to be helped along by hand, and then it definitely
helps to know more about how git works.

Anyway, the quick tutorial about "git bisect" is that once you've given it
the required first "good" and "bad" points, it will create a new branch in
the repository (called "bisect", in case you care), and after that point
it will do a search in the commit DAG (aka "history tree" - it's not a
tree, it's a DAG, since merges will join branches together) for the next
commit that will neatly "split" the DAG into two equal pieces. It will
keep splitting the commit history until you get fed up, or until it has
pinpointed the single commit that caused the problem.

The nicest tool to use during bisection is to just do a

	git bisect visualize

that simply starts up "gitk" (the default git history visualizer) to show
what the current state of bisection is. Now, if there are thousands and
thousands of commits, you'll have a really hard time getting a visual clue
about what is going on, but especially once you get to a smaller set of
commits, it's very useful indeed.

And it's _especially_ useful if you hit one of the problem spots where you
can't test the resulting tree for some unrelated reason. When that
happens, you should _not_ mark the problematic commit as being "bad",
because you really don't know - the "badness" of that commit is probably
not related to the "badness" that you're actually searching for.

Instead, you should say "ok, I refuse to test this commit at all, because
it's got other problems, and I will select another commit instead". The
bisection algorithm doesn't care which commit you pick, as long as it's
within the set of "unknown" commits that you'll see with the visualization
tool.

Of course, for efficiency reasons, the _closer_ you get to the half-way
mark, the better. So it's useful to try to pick a commit that is close to
the one that "git bisect" originally chose for you, but that's not a
correctness issue, that's just an issue of "if we have a thousand
potential commits, we're better off bisecting it 400/600 rather than
1/999, even if the exact half-way point isn't testable".

So if you need to decide to pick another point than the one "git bisect"
chose for you automatically, just select that commit in the visualizer
(which will cut the SHA1 name of it), and then do

	git reset --hard <paste-sha1-here>

to reset the "bisect" branch to that point instead. And then compile and
test that kernel instead (and then if that's good or bad, you can do the
"git bisect good" or "git bisect bad" thing to mark it so, and git will
continue to bisect the set of commits).

It can be a bit boring, but damn, it's effective. I've used "git bisect"
several times when I've been too lazy to try to really think about what is
going on - I'll happily brute-force bug-finding even if it might take a
little longer, if it's guaranteed to find it (and if the bug is
reproducible, git bisect definitely guarantees to find what made it
appear, even if that may not necessarily be the deeper _cause_ of the bug)

		Linus

From: Linus Torvalds <torvalds@linux-foundation.org>
Newsgroups: fa.linux.kernel
Subject: Re: 2.6.21-rc1: known regressions (part 2)
Date: Thu, 01 Mar 2007 23:40:46 UTC
Message-ID: <fa.63Ct4ClRD8hjd7Z3taISujCscb8@ifi.uio.no>

On Thu, 1 Mar 2007, Ingo Molnar wrote:
>
> * Ingo Molnar <mingo@elte.hu> wrote:
>
> > update: f3ccb06f3b8e0cf42b579db21f3ca7f17fcc3f38 works for me too, and
> > 01363220f5d23ef68276db8974e46a502e43d01d is broken. I too will attempt
> > to bisect this.
>
> hm. There's some weird bisection artifact here. Here are the commits i
> tested, in git-log order:
>
> #1 commit 01363220f5d23ef68276db8974e46a502e43d01d bad
> #2 commit ee404566f97f9254433399fbbcfa05390c7c55f7 bad
> #3 commit f3ccb06f3b8e0cf42b579db21f3ca7f17fcc3f38 good
> #4 commit c827ba4cb49a30ce581201fd0ba2be77cde412c7 bad

Use "git bisect visualize" to see what bisect ends up doing.

> if i tell git-bisect that #1 is bad and #3 is good, then it offers me #2
> - that's OK. But when i tell it that #2 is bad, it offers #4 - which is
> out of order!

No it's not. "git bisect" does exactly the right thing. There is no simple
ordering in a complex branch-merge scenario, you can't just put the
commits in some "ordering" and test things in time order. That would be
totally broken, and idiotic. It doesn't give the right results.

What git bisect does is to find the commit that most closely *bisects* the
history of commits, so that if it is marked good/bad, it will leave you
with about 50% of the commits left. But if you are looking at date order,
you're entirely confused.

For example, let's take a really simple case

	    a <- bad
	   / \
          b   c
	  |   |
	  d   e
	  |   |
	  f   g
	   \ /
	    h
            |
	    * <-good

and if you are looking to find something "in the middle", you might thing
that "d" or "e" are the best choices, since time-wise, they are in the
middle.

But that's not true AT ALL.

If you actually want to bisect that kind of history, you need to choose
"b" or "c", even though they may both be *much* more "recent" than the
others. Why? Because if you pick "d", you're really only testing three
commits ('d' 'f' and 'h') out of the 8 commits you have to test.

In contrast, if you pick 'b', you are testing the effects of *four*
commits ('b', 'd', 'f' and 'h') and you have thus neatly bisected the
commits into two equal groups for testing (one group _with_ those four
commits, and one group _without_) instead of having partitioned them as 3
commits vs 5 commits.

So please realize that non-linear history very much means that you MUST
NOT think that you just pick a commit "in the middle". No, git bisect is a
LOT smarter than that - it picks a commit that *reaches* about half the
commits you have left to test.

> The bisection goes off into la-la land after that and
> never gets back to a commit that is /after/ the good commit. How is this
> possible? (I upgraded from git-1.4.4 to 1.5.0 to make sure this isnt
> some git bug that's already fixed.)

It's possible because git knows what it is doing, and you didn't think
things through.

The commits that "git bisect" picked out are the right ones. Quite often,
there may be two or more "equally good" commits (in my example above, you
can choose either "b" or "c", and it will bisect the set of untested
commits equally well - in two groups of four, but two *different* groups
of four commits), and yes, it's possible that git has a bug that makes it
pick the wrong ones, but quite frankly, I seriously doubt it. "git bisect"
has been very successful indeed, and is generally a *lot* better at
picking a commit "in the middle" than people are, exactly because it's
quite hard to see which commit "reaches" half the commits if you have lots
of merges and branches.

Try out

	git bisect visualize

and it will literally show you what it is doing.

What can be confusing is that if the "good" and "bad" markers are ON
DIFFERENT BRANCHES OF DEVELOPMENT, you may not even *see* the "good"
marker, because you may well have something like this:


	a <- bad
	|
	b   * <- good
	|   |
	c   d
	 \ /
	  e
	  |
	  f
	  |
	 ...

and what do you think "git bisect visualize" will actually show you?

Since 'd', 'e' and 'f' are all in the "good" set (they both exist as
commits in something leading up to a commit that has already been deemed
fine), they aren't *interesting* - they can't be introducing the bug,
since if that was the case, the good commit wouldn't have been good. So as
far as bisection is concerned, the tree actually looks like

	 a <- bad
	 |
	 b
	 |
	 c
	 |
	...

and you have just three commits that are potentially interesting: 'a', 'b'
and 'c'.

Now, with three commits, you cannot test them half-and-half, so you have
to test it in groups of 1 vs 2 commits, so it's arbitrary whether you
choose 'b' or 'c' to test, but you'd test one of them. Say that you choose
'b', and it turns out to be good. If so, you're done: 'a' is bad and 'b'
is good, so the bug was introduced in 'a'. But if it turns out to be bad,
you'll still have to test 'c' too, since you don't know if the bug was
*introduced* in 'b' or not.

See?

> i'll try to straighten this out manually

Don't. You're just going to make your bisection much less effective. The
whole point of bisection is that you can usually cut the number of commits
to test pretty exactly in half.  If you start mucking with the commits to
test, and you don't understand about the reachability graph, you'll just
choose a much worse set of commits to test than "git bisect" will do.

So learn to trust "git bisect". It really does know what it is doing.

		Linus

From: Linus Torvalds <torvalds@linux-foundation.org>
Newsgroups: fa.linux.kernel
Subject: Re: 2.6.21-rc1: known regressions (part 2)
Date: Fri, 02 Mar 2007 00:31:00 UTC
Message-ID: <fa.NTa0foc6yshUr2H9+kfHCLAmL60@ifi.uio.no>

On Thu, 1 Mar 2007, Ingo Molnar wrote:
>
> git-bisect gets royally confused on those ACPI merge branches around
> commit c0cd79d11412969b6b8fa1624cdc1277db82e2fe. Here are my test
> results so far:

Looks like git bisect worked for you, and wasn't confused at all. You
started out with 2931 commits between your first known-bad and known-good
commits, which means that you usually end up having to check "log2(n)+1"
kernels, ie I'd have expected you to have to do 12-13 bisection attempts
to cut it down to one.

You seem to have done 14 (you list 16 commits, two of which are the
starting points), which is right in that range. The reason you sometimes
get more is:

 - you "help" git bisect by choosing other commits than the optimal ones.

 - with bad luck, it can be hard to get really close to "half the commits"
   in the reachability analysis, especially if you have lots of merges
   (and *especially* if you have octopus merges that merge more than two
   branches of development). For example, say that you have something like

	           a
	           |
	   +---+---+---+---+
	   |   |   |   |   |
	   b   c   d   e   f

   where you have six commits - you can't test any "combinations" at all,
   since they are all independent, so "git bisect" cannot test them three
   and three to cut down the time, so if you don't know which one is bad,
   you'll basically end up testing them all.

The bad luck case never really happens to that extreme in practice, and
even when it does you can sometimes be lucky and just hit on the bug early
(so "bad luck" may end up being "good luck" after all), but it explains
why you can get more - or less - than log2(n)+1 attempts. More commonly
one more.

A much *bigger* problem is if you mark something good or bad that isn't
really. Ie if the bug comes and goes (it might be timing-dependent, for
example), the problem will be that you'll always narrow things down
(that's what bisection does), but you may not narrow it down to the right
thing!

We've had that happen several times. If the bug (for example) means that
suspend *often* breaks, but sometimes works just by luck, you might mark a
kernel "good" when it really wasn't and then "git bisect" will *really* go
out in the weeds, and won't even try to test the commits that may have
introduced the bug, because you told it that those commits resulted in a
good kernel..

>  commit 01363220f5d23ef68276db8974e46a502e43d01d: bad
>  commit 255f0385c8e0d6b9005c0e09fffb5bd852f3b506: bad
>  commit c0cd79d11412969b6b8fa1624cdc1277db82e2fe: bad
>  commit c24e912b61b1ab2301c59777134194066b06465c: good
>  commit e9e2cdb412412326c4827fc78ba27f410d837e6e: bad
>  commit 79bf2bb335b85db25d27421c798595a2fa2a0e82: bad
>  commit fc955f670c0a66aca965605dae797e747b2bef7d: good
>  commit 70c0846e430881967776582e13aefb81407919f1: good
>  commit 414f827c46973ba39320cfb43feb55a0eeb9b4e8: bad
>  commit f3ccb06f3b8e0cf42b579db21f3ca7f17fcc3f38: good
>  commit 5f0b1437e0708772b6fecae5900c01c3b5f9b512: bad
>  commit b878ca5d37953ad1c4578b225a13a3c3e7e743b7: bad
>  commit c2902c8ae06762d941fab64198467f78cab6f8cd: bad
>  commit 12e74f7d430655f541b85018ea62bcd669094bd7: bad
>  commit 3388c37e04ec0e35ebc1b4c732fdefc9ea938f3b: bad
>  commit 9f4bd5dde81b5cb94e4f52f2f05825aa0422f1ff: bad

Looks like it's claiming that 9f4bd5dde81b5cb94e4f52f2f05825aa0422f1ff is
the bad commit. Which is extremely unlikely, since it only seems to affect
the emu10k sound driver, which I don't think even exists on any ThinkPad
laptops (correct me if I'm wrong).

Btw, you seem to have re-ordered the commits - the above is not the order
you did the bisection in. The known-good commit (f3ccb06..) is in the
middle. That's totally bogus. Please use the git bisection log (see
git/BISECT_LOG), and don't think that you know some "better" order. You
really don't.

> the results are totally reproducible (i re-tried a few of both the good
> and the bad commits), i.e. it's not a sporadic condition. Also, a number
> of the 'bad' commits have no dynticks stuff in them at all, so i'd
> exclude dynticks.
>
> could someone suggest a sane way to go with this? Perhaps suggest
> specific commit IDs to test?

You claim that 9f4bd5dd is bad, but you indirectly claim that its direct
parent (5986a2ec) is good by saying that f3ccb06f is good. This is why
"git bisect" will claim that 9f4bd5dd must be the bad commit.

I would suggest testing commit 5986a2ec explicitly. If that one is good,
then, since you claim that 9f4bd5dd is bad, then yes, 9f4bd5dd *is* the
bad commit (because 5986a2ec is its direct parent).

But most likely, 9f4bd5dd is actually already bad, and what you are seeing
is two *different* bugs that just have the same symptoms ("suspend doesn't
work").

What happens is that you've chased them *both*, and you cannot bisect that
kind of behaviour totally automatically and mindlessly, simply because
when you say "git bisect bad", that means that *one* of the bugs is
active, but not necessarily both of them. So you may well be marking
kernels that are "good" (as far as the other bug is concerned) as bad -
and that just means that bisection won't even test them.

When that happens, you need to basically

 - be able to separate the bugs out some way (so that you can still mark a
   non-working kernel "good" if it's good *with*respect*to* the particular
   bug you're chasing)

   This is often practically impossible, _especially_ with suspend, where
   the behaviour is so unhelpful that it's usually not possible to
   separate out "ACPI is broken" from "one particular device driver is
   broken", because they both have exactly the same symptoms: the machine
   doesn't resume.

HOWEVER. Even if you can't actually separate the bugs out, you can usually
find where *one* of the bugs starts, and that point you can generally find
the fix for it too. In this case, we already know one of the bugs: it's
the ACPI bug that was apparently fixed by f3ccb06f3 (or maybe another one
in that series).

Once you have that, you now actually have a way to "correct" for that
known bug, and by correcting for the known bug, you now *can* separate the
behaviour of the two bugs:

 - You can now re-do a totally mindless git bisection for the *other* bug,
   but what you now need to do is that at each bisection step, you look at
   whether the bisection point has the known bug, and if so, you apply the
   known fix for that known bug, and thus you can test the kernel
   *without* the interaction of the bug you already found.

This makes bisection a lot less automated (you have to apply the "fix" for
the other bug at each step), but it still allows "total automation" in the
sense that you don't actually need to know at all what you're looking for:
you're just following a known pattern, and you're basically just
correcting for the effects of another bug that you're no longer interested
in, since you already know what the fix for that bug was.

The other alternative is to actually have a clue what you're searching
for, and/or look deeply at where the fix was merged, and trying to narrow
things down by actually understanding the problem. But at that point,
bisection won't much help you, except perhaps as a way to find a mid-way
point to test out theories with ("which drivers that I actually use have
changed in between" kinds of experiments where you simply undo part of
the changes entirely, and bisection ends up being just a way to pick
points that are hopefully "interestingly far apart").

			Linus

From: Linus Torvalds <torvalds@linux-foundation.org>
Newsgroups: fa.linux.kernel
Subject: Re: 2.6.21-rc1: known regressions (part 2)
Date: Fri, 02 Mar 2007 00:45:25 UTC
Message-ID: <fa.8mGUFyTdx8CqCcb1RxAuXU3++W0@ifi.uio.no>

On Thu, 1 Mar 2007, Linus Torvalds wrote:
>
> Once you have that, you now actually have a way to "correct" for that
> known bug, and by correcting for the known bug, you now *can* separate the
> behaviour of the two bugs:
>
>  - You can now re-do a totally mindless git bisection for the *other* bug,
>    but what you now need to do is that at each bisection step, you look at
>    whether the bisection point has the known bug, and if so, you apply the
>    known fix for that known bug, and thus you can test the kernel
>    *without* the interaction of the bug you already found.
>
> This makes bisection a lot less automated (you have to apply the "fix" for
> the other bug at each step)

Side note: it's still usually fairly easy. Especially if you have a known
fix for the other bug, you can usually just do the equivalent of

	git cherry-pick <fixcommit>

at each point during this bisect (or just have a known patch that you keep
applying and un-applying), and you're largely done.

Of course, if the area with the fix keeps changing, or if the fix is
really intrusive and nasty, this gets hairy, but at least in this case the
patch is fairly trivial and it shouldn't cause any trouble at all to do
this.

The only real down-side is just the mindless extra work, and the possible
added confusion you get from modifying the history at the points you're
testing. "git bisect" is not necessarily happy about auto-picking a new
bisection point with a dirty tree, for example, so before you mark
something "good" or "bad", you should generally try to do so with a clean
git tree (ie if you apply a patch at each stage, do "git reset --hard" to
remove the patch before you do the "git bisect bad/good" stage).

Similarly, especially at the end of the bisection run, if you actually use
"git cherry-pick" to *add* a commit, the bisection will now take that
added commit into account when trying to pick the next commit to check,
which is not what you really want. It probably doesn't matter that much
during the early stages (when bisection is really jumping around wildly
anyway, and one commit more or less doesn't really matter), but again, it
might be a good idea to make a habit of undoing the cherry-pick, the same
way you'd undo a patch (eg "git reset --hard HEAD^" would do it, if you
have exactly one cherry-pick that you tested).

		Linus

From: Linus Torvalds <torvalds@linux-foundation.org>
Newsgroups: fa.linux.kernel
Subject: Re: Regression: USB is nfg after suspend/resume(RAM) cycle on Intel
Date: Tue, 29 May 2007 17:42:48 UTC
Message-ID: <fa.zWzTluJWEfNpa88JpCctkkanoBA@ifi.uio.no>

On Tue, 29 May 2007, Mark Lord wrote:
>
> Ugh.  Is there a way to tell bisect to only work around the USB updates?

Well, you _can_ actually give "git bisect" a pathspec (the same way you
git "git log" and friends), and tell it to only care about stuff that
changed that pathspec.

However, that was broken in some older git versions, and in general hasn't
had a huge amount of testing even in new ones, so I'm a tad nervous about
recommending people do it. But yes, you should be able to say

	git bisect start drivers/usb/
	git bisect good v2.6.21
	git bisect bad v2.6.22-rc3

and off you go. However, if you do this, you need to make sure that you
have at LEAST git-1.5.1.

The other downside of path limiting is that if it turns out that the bug
was really introduced by something else, and just happened to _look_ like
it's USB-related, the path limiting will then cause "git bisect" to blame
a commit that just happens to be the next commit after the _real_ bug was
introduced.

In this case, I don't think it's likely to be an issue, but it *could*
obviously be something else.

(In contrast, the non-path-limited "git bisect" should work in pretty much
any situation and with any git version)

> Still, that'll take a few hours, and frankly I'm getting sick of having
> to re-debug the USB layer with each new kernel rev.

Yeah, I'm not surprised. USB is probably the worst possible case for
suspend/resume (at least if you ignore ACPI-related problems). It's nasty.

> Got a pointer to the "bisect how-to" ?

It's so disgustingly simple that I don't think we've ever done any
specific bisection tutorial, but the "git-bisect" man-page does exist, and
it talks about the only half-way interesting case, namely the case where
the automatic selection of a half-way point causes git to pick a point
that doesn't work for some other reason (ie stupid compile problem or
whatever). In which case you have to pick another one manually.

So that kind of gotcha is at least _mentioned_ in the git-bisect man-page,
even if it doesn't get much further than that.

There's also the git users manual, but I think the man-page is more
detailed. But for future reference, just do

	git user manual

in google, and press "I'm feeling lucky". It finds the right thing at
least for me (and at least right now).

			Linus

From: Linus Torvalds <torvalds@linux-foundation.org>
Newsgroups: fa.linux.kernel
Subject: Re: Linux 2.6.22 released
Date: Tue, 10 Jul 2007 15:39:56 UTC
Message-ID: <fa.DEwDXyqrl1AOg838sXIQpMmK16U@ifi.uio.no>

On Tue, 10 Jul 2007, Stefano Rivoir wrote:
>
> 2.6.22 hangs at boot on my box. Here attached a original dmesg from 2.6.21,
> and a copy of it where it stops on 2.6.22 (I can't attach the original 2.6.22
> dmesg because it's not logged to disk yet); it actually stops right after
> 'init' launches.

Ok, without any oops or hang in any really obvious place, that doesn't
really tell anybody anything specific enough to even start trying to debug
this, so you'd need to do one of two things:

 - poke a lot at the machine to try to get more specific information. In
   particular, get things like SysRQ-T output. That, in turn, probably
   would mean trying to get a serial console hooked up or something.

   The next thing that you got on 2.6.21 after the point it hangs was
   apparently

	..
	input: PC Speaker as /class/input/input1
	ieee1394: Initialized config rom entry `ip1394'
	ACPI: PCI Interrupt Link [APC4] enabled at IRQ 19
	ACPI: PCI Interrupt 0000:04:05.0[A] -> Link [APC4] -> GSI 19 (level, low) -> IRQ 19
	usbcore: registered new interface driver hiddev
	..

   so you could _try_ to disable the PC speaker or firewire, and see
   what's up. Did you switch from the old firewire drivers to the new one,
   for example? Or if you didn't, try it.

   IOW, we'd need a lot more debug information.

The second alternative will take some time, but is really a lot easier:

 - Get a kernel git tree, and do a "git bisect".

   There's almost 7000 commits in between 2.6.21 and 22, but that still
   means that in about fourteen recompiles/reboots, "git bisect" should
   tell us where your problem starts, which will hopefully make it obvious
   what the problem is (or at least pinpoint it a *lot*).

Doing a git bisect isn't really that hard, but fourteen compiles/reboots
will take some time (well, the compiles will, the reboots aren't that
bad). But even if you're not a git user, it really is very simple:

 - get started with 'git': on most distros it's now as simple as doing
   something like

	yum install git

   and while you might not get the latest version (Debian stable is at
   some ancient 1.4.4.4 version that isn't as nice as the 1.5.x series),
   for something like this you won't care that deeply.

 - get the kernel git tree (this will take a while to download about
   180MB)

	git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6

 - start the "git bisect" with

	git bisect good v2.6.21
	git bisect bad v2.6.22

   and it will pick a kernel version about half-way between the two
   points, and you can now start testing. For each kernel you try, if it
   boots fine, do "git bisect good", otherwise boot into a working kernel,
   and then do "git bisect bad". Git will then pick the next "halfway"
   kernel for that case.

Thanks,

		Linus

From: Linus Torvalds <torvalds@linux-foundation.org>
Newsgroups: fa.linux.kernel
Subject: Re: GIT bisection range errors
Date: Thu, 08 May 2008 23:00:39 UTC
Message-ID: <fa.lMeHf8njqvCZ0Y4BjVIQtc3VesI@ifi.uio.no>

On Fri, 9 May 2008, Rene Herman wrote:
>
> I'm in a git bisect and am experiencing strangeness. I did a
>
> $ git checkout -b rc v2.6.26-rc1
> $ git bisect start
> $ git bisect bad
> $ git bisect good v2.6.25
>
> Yet, during this I'm finding myself at 2.6.25-rc6 and 2.6.25-rc8
> as the last two results (both good...).

This is very normal.

Why?

Because a lot (in fact, *most*) of the code that was merged after v2.6.25
was released was actually *written* and committed long before v2.6.25.

It just got merged into my tree much later.

So what happens? The bisection run starts walking into all that history,
and that history is *not* based on the released v2.6.25 at all, it's based
on much earlier kernels (eg the -rc kernels).

So what you see is perfectly normal and expected. It's only unexpected if
you think of history as a linear thing, but it isn't - it's full of
merging of code that was branched off from (much) earlier code points.

		Linus