Error

Unrecognized LW server error:

Field "fmCrosspost" of type "CrosspostOutput" must have a selection of subfields. Did you mean "fmCrosspost { ... }"?

Rohin Shah 21 Feb 2023 18:45 UTC
37 points
6 ∶ 3
Your post argues for a strong conclusion:
To spot the error in these arguments, we only have to look up what cited ‘coherence theorems’ actually say. And yet the error seems to have gone uncorrected for more than a decade.
[...]
There are no coherence theorems. Authors in the AI safety community should stop suggesting that there are.
There are money-pump arguments, but the conclusions of these arguments are not theorems. The arguments depend on substantive and doubtful assumptions.
As I understand it, you propose two main arguments for the conclusion:
1. There are only arguments about money-pumps / dominated strategies, not theorems.
2. The Completeness axiom is suspicious.
I think (1) is straightforwardly wrong / conceptually confused. I agree with skepticism on the basis of (2), but people have already noticed this and discussed it (though phrased differently).
(Note the post makes other smaller claims that I either disagree with or think are misleading—don’t assume that if I don’t talk about some claim that means I think it’s correct.)
For the first argument, that there are no “coherence theorems”:
For now, the important thing to note is that the conclusions of money-pump arguments are not theorems. Theorems (like the VNM Theorem) can be proved without making any substantive assumptions. Money-pump arguments establish their conclusion only by making substantive assumptions: assumptions that might well be false. [...]
That is definitely not the difference between theorems and arguments. Theorems are typically of the form “Suppose X, then Y”; what is X if not an assumption?
For example, in (one direction of) the VNM theorem, the assumption is that the preferences satisfy transitivity, completeness, independence, and continuity, and the conclusion is that the preferences can be represented with a utility function.
The difference between theorems and arguments is that in theorems you are limited to a particular set of formal inference rules in moving from premises to conclusions, whereas in arguments there is a much more expansive and informal set of inference rules. (Though in practice people use informal arguments in proving theorems with the implicit promise that they could be rewritten with the formal inference rules with more effort.)
In any case, if you really want to see one, here’s a fairly boring money-pump theorem / coherence theorem:
Theorem. Suppose there is a set of possible worlds $W = {w^{1}, \dots w^{N}}$ , and an agent $A : W \times W \to R$ that given a current world $w_{i}$ and a proposed new world $w_{j}$ specifies how much money it would pay to switch to $w_{j}$ from $w_{i}$ . Suppose further than $A$ cannot be money pumped, that is, there is no sequence of worlds $w_{1}, w_{2}, \dots w_{k}$ such that (1) $w_{1} = w_{k}$ and (2) $k - 1 \sum i = 1 A (w_{i}, w_{i + 1}) > 0$ . Then $A$ must be transitive in the following sense: for any $w_{1}, w_{2}, w_{3}$ , if $A (w_{1}, w_{2}) > 0$ and $A (w_{2}, w_{3}) > 0$ , then $A (w_{3}, w_{1}) < 0$ .
Proof. Suppose $A$ is not transitive, so there exists some $w_{1}, w_{2}, w_{3}$ where $A (w_{1}, w_{2}) > 0$ , $A (w_{2}, w_{3}) > 0$ , and $A (w_{3}, w_{1}) \geq 0$ . But then then the sequence $[w_{1}, w_{2}, w_{3}, w_{1}]$ is a money pump, leading to a contradiction.
This theorem is baking in some assumptions that you might find problematic, such as completeness (implicitly present in the type signature of $A$ ), or “no money pumps” (which you might object to because there’s no one to actually run the money pump on the agent), or the lack of time-dependence of the agent (again implicitly present in the type signature of $A$ ).
But I think this is clearly a theorem that is coming to a substantive conclusion about an agent based on “no dominated strategies” / “no money pumps”, so I don’t think you can really say that “coherence theorems don’t exist”.
For the second argument (that the completeness axiom is suspicious): I think this is basically expressing the same sort of objection that I express here, particularly the section “There are no coherence arguments that say you must have preferences”. I didn’t tie it to the Completeness axiom because I think it’s a mistake to get bogged down in the details of the specific assumptions present in theorems when you can make the same point in English, but it is the same conceptual point, as far as I can tell.
For what it’s worth my position here is “you can’t argue for AI risk solely via coherence theorems; you also have to argue for why the AI will be goal-directed in the first place, but there are plausible arguments for that conclusion (which are not based on coherence arguments)”.
- EJT 21 Feb 2023 23:20 UTC
  17 points
  5 ∶ 0
  Parent
  Theorems are typically of the form “Suppose X, then Y”; what is X if not an assumption?
  X is an antecedent.
  Consider an example. Imagine I claim:
  - Suppose James is a bachelor. Then James is unmarried.
  In making this claim, I am not assuming that James is a bachelor. My claim is true whether or not James is a bachelor.
  I might temporarily assume that James is a bachelor, and then use that assumption to prove that James is unmarried. But when I conclude ‘Suppose James is a bachelor. Then James is unmarried’, I discharge that initial assumption. My conclusion no longer depends on it. Any conclusion which can be proved with no undischarged assumptions is a theorem.
  Theorem. Suppose there is a set of possible worlds $W = {w^{1}, \dots w^{N}}$ , and an agent $A : W \times W \to R$ that given a current world $w_{i}$ and a proposed new world $w_{j}$ specifies how much money it would pay to switch to $w_{j}$ from $w_{i}$ . Suppose further than $A$ cannot be money pumped, that is, there is no sequence of worlds $w_{1}, w_{2}, \dots w_{k}$ such that (1) $w_{1} = w_{k}$ and (2) $k - 1 \sum i = 1 A (w_{i}, w_{i + 1}) > 0$ . Then $A$ must be transitive in the following sense: for any $w_{1}, w_{2}, w_{3}$ , if $A (w_{1}, w_{2}) > 0$ and $A (w_{2}, w_{3}) > 0$ , then $A (w_{3}, w_{1}) < 0$ .
  Proof. Suppose $A$ is not transitive, so there exists some $w_{1}, w_{2}, w_{3}$ where $A (w_{1}, w_{2}) > 0$ , $A (w_{2}, w_{3}) > 0$ , and $A (w_{3}, w_{1}) \geq 0$ . But then then the sequence $[w_{1}, w_{2}, w_{3}, w_{1}]$ is a money pump, leading to a contradiction.
  I agree that this is a theorem. But it’s not a ‘coherence theorem’ (at least not in the way that I’ve used the term in this post, and not in the way that previous authors seem to have used the term [see the Appendix]): it doesn’t state that, unless an agent can be represented as maximizing expected utility, that agent is liable to pursue strategies that are dominated by some other available strategy. It states only that, unless an agent’s preferences are acyclic, that agent is liable to pursue strategies that are dominated by some other available strategy.
  You can call it a ‘coherence theorem’. Then it would be true that coherence theorems exist. But the important point remains: Premise 1 of the coherence argument is false. There are no theorems which state that, unless an agent can be represented as maximizing expected utility, that agent is liable to pursue strategies that are dominated by some other available strategy. VNM doesn’t say that, Savage doesn’t say that, Bolker-Jeffrey doesn’t say that, Dutch Books don’t say that, Cox doesn’t say that, Complete Class doesn’t say that.
  For the second argument (that the completeness axiom is suspicious): I think this is basically expressing the same sort of objection that I express here, particularly the section “There are no coherence arguments that say you must have preferences”. I didn’t tie it to the Completeness axiom because I think it’s a mistake to get bogged down in the details of the specific assumptions present in theorems when you can make the same point in English, but it is the same conceptual point, as far as I can tell.
  I agree. I think the points that you make in that post are good.
  For what it’s worth my position here is “you can’t argue for AI risk solely via coherence theorems; you also have to argue for why the AI will be goal-directed in the first place, but there are plausible arguments for that conclusion (which are not based on coherence arguments)”.
  I agree with this too.
  - Rohin Shah 22 Feb 2023 0:19 UTC
    12 points
    5 ∶ 1
    Parent
    Thanks, I understand better what you’re trying to argue.
    The part I hadn’t understood was that, according to your definition, a “coherence theorem” has to (a) only rely on antecedents of the form “no dominated strategies” and (b) conclude that the agent is representable by a utility function. I agree that on this definition there are no coherence theorems. I still think it’s not a great pedagogical or rhetorical move, because the definition is pretty weird.
    I still disagree with your claim that people haven’t made this critique before.
    From your discussion:
    [The Complete Class Theorem] does refer to dominated strategies. However, the Complete Class Theorem starts off by assuming that the agent’s preferences over actions in sets of circumstances satisfy Completeness and Transitivity. If the agent’s preferences are not complete and transitive, the Complete Class Theorem does not apply. So, the Complete Class Theorem does not imply that agents must be representable as maximizing expected utility if they are to avoid pursuing dominated strategies.
    So, you would agree that the following is an English description of a theorem:
    If an agent has complete, transitive preferences, and it does not pursue dominated strategies, then it must be representable as maximizing expected utility.
    The difference from your premise 1 is the part about the agent having complete, transitive preferences.
    I feel pretty fine with justifying the transitive part via theorems basically like the one I gave above. You’d need to strengthen it a bit but that seems very doable. You do require a money pump argument rather than a dominated strategy argument, because when you have intransitive preferences it’s not even clear what a “dominated strategy” would be.
    If you buy that, then the only difference is the part about the agent having complete preferences. Which is exactly what has been critiqued previously. So I still think that it is basically incorrect to say:
    And yet the error seems to have gone uncorrected for more than a decade.
    - EJT 22 Feb 2023 1:13 UTC
      8 points
      2 ∶ 0
      Parent
      So, you would agree that the following is an English description of a theorem:
      If an agent has complete, transitive preferences, and it does not pursue dominated strategies, then it must be representable as maximizing expected utility.
      Yep, I agree with that.
      I feel pretty fine with justifying the transitive part via theorems basically like the one I gave above.
      Note that your money-pump justifies acyclicity (The agent does not strictly prefer A to B, B to C, and C to A) rather than the version of transitivity necessary for the VNM and Complete Class theorems (If the agent weakly prefers A to B, and B to C, then the agent weakly prefers A to C). Gustafsson thinks you need Completeness to get a money-pump for this version of transitivity working (see footnote 8 on page 3), and I’m inclined to agree.
      when you have intransitive preferences it’s not even clear what a “dominated strategy” would be.
      A dominated strategy would be a strategy which leads you to choose an option that is worse in some respect than another available option and not better than that other available option in any respect. For example, making all the trades and getting A- in the decision-situation below would be a dominated strategy, since you could have made no trades and got A:
      So I still think that it is basically incorrect to say:
      And yet the error seems to have gone uncorrected for more than a decade.
      The error is claiming that
      There exist theorems which state that, unless an agent can be represented as maximizing expected utility, that agent is liable to pursue strategies that are dominated by some other available strategy.
      I haven’t seen anyone point out that that claim is false.
      That said, one could reason as follows:
      Rohin, John, and others have argued that agents with incomplete preferences can act in accordance with policies that make them immune to pursuing dominated strategies.
      Agents with incomplete preferences cannot be represented as maximizing expected utility.
      So, if Rohin’s, John’s, and others’ arguments are sound, there cannot exist theorems which state that, unless an agent can be represented as maximizing expected utility, that agent is liable to pursue strategies that are dominated by some other available strategy.
      Then one would have corrected the error. But since the availability of this kind of reasoning is easily missed, it seems worth correcting the error directly.
      - Rohin Shah 22 Feb 2023 15:34 UTC
        5 points
        3 ∶ 1
        Parent
        Okay, it seems like we agree on the object-level facts, and what’s left is a disagreement about whether people have been making a major error. I’m less interested in that disagreement so probably won’t get into a detailed discussion, but I’ll briefly outline my position here.
        The error is claiming that
        There exist theorems which state that, unless an agent can be represented as maximizing expected utility, that agent is liable to pursue strategies that are dominated by some other available strategy.
        I haven’t seen anyone point out that that claim is false.
        The main way in which this claim is false (on your way of using words) is that it fails to note some of the antecedents in the theorem (completeness, maybe transitivity).
        But I don’t think this is a reasonable way to use words, and I don’t think it’s reasonable to read the quotes in your appendix as claiming what you say they claim.
        Converting math into English is a tricky business. Often a lot of the important “assumptions” in a theorem are baked into things like the type signature of a particular variable or the definitions of some key terms; in my toy theorem above I give two examples (completeness and lack of time-dependence). You are going to lose some information about what the theorem says when you convert it from math to English; an author’s job is to communicate the “important” parts of the theorem (e.g. the conclusion, any antecedents that the reader may not agree with, implications of the type signature that limit the applicability of the conclusion), which will depend on the audience.
        As a result when you read an English description of a theorem, you should not expect it to state every antecedent. So it seems unreasonable to me to critique a claim in English about a theorem existing purely because it didn’t list all the antecedents.
        I think it is reasonable to critique a claim in English about a theorem on the basis that it didn’t highlight an important antecedent that limits its applicability. If you said “AI alignment researchers should make sure to highlight the Completeness axiom when discussing coherence theorems” I’d be much more sympathetic (though personally my advice would be “AI alignment researchers should make sure to either argue for or highlight as an assumption the point that the AI is goal-directed / has preferences”).
        Gustafsson thinks you need Completeness to get a money-pump for this version of transitivity working
        Yup, good point, I think it doesn’t change the conclusion.
        EJT 22 Feb 2023 23:13 UTC
        5 points
        2 ∶ 0
        Parent
        it seems like we agree on the object-level facts
        I think that’s right.
        Often a lot of the important “assumptions” in a theorem are baked into things like the type signature of a particular variable or the definitions of some key terms; in my toy theorem above I give two examples (completeness and lack of time-dependence). You are going to lose some information about what the theorem says when you convert it from math to English; an author’s job is to communicate the “important” parts of the theorem (e.g. the conclusion, any antecedents that the reader may not agree with, implications of the type signature that limit the applicability of the conclusion), which will depend on the audience.
        Yep, I agree with all of this.
        Converting math into English is a tricky business.
        Often, but not in this case. If authors understood the above points and meant to refer to the Complete Class Theorem, they need only have said:
        If an agent has complete, transitive preferences, and it does not pursue dominated strategies, then it must be representable as maximizing expected utility.
        (And they probably wouldn’t have mentioned Cox, Savage, etc.)
        Yup, good point, I think it doesn’t change the conclusion.
        I think it does. If the money-pump for transitivity needs Completeness, and Completeness is doubtful, then the money-pump for transitivity is doubtful too.
        Rohin Shah 23 Feb 2023 15:29 UTC
        3 points
        1 ∶ 0
        Parent
        I think it does [change the conclusion].
        Upon rereading I realize I didn’t state this explicitly, but my conclusion was the following:
        If an agent has complete preferences, and it does not pursue dominated strategies, then it must be representable as maximizing expected utility.
        Transitivity depending on completeness doesn’t invalidate that conclusion.
        EJT 23 Feb 2023 18:13 UTC
        3 points
        0 ∶ 0
        Parent
        Ah I see! Yep, agree with that.
NunoSempere 21 Feb 2023 0:16 UTC
26 points
13 ∶ 7
I appreciate the whole post. But I personally really enjoyed the appendix. In particular, I found it informative that Yudkowsk can speak/write with that level of authoritativeness, confidence, and disdain for others who disagree, and still be wrong (if this post is right).
- Habryka [Deactivated] 21 Feb 2023 0:59 UTC
  10 points
  10 ∶ 10
  Parent
  (if this post is right)
  The post does actually seem wrong though.
  I expect someone to write a comment with the details at some point (I am pretty busy right now, so can only give a quick meta-level gleam), but mostly, I feel like in order to argue that something is wrong with these arguments is that you have to argue more compellingly against completeness and possible alternative ways to establish dutch-book arguments.
  Also, the title of “there are no coherence arguments” is just straightforwardly wrong. The theorems cited are of course real theorems, they are relevant to agents acting with a certain kind of coherence, and I don’t really understand the semantic argument that is happening where it’s trying to say that the cited theorems aren’t talking about “coherence”, when like, they clearly are.
  You can argue that the theorems are wrong, or that the explicit assumptions of the theorems don’t hold, which many people have done, but like, there are still coherence theorems, and IMO completeness seems quite reasonable to me and the argument here seems very weak (and I would urge the author to create an actual concrete situation that doesn’t seem very dumb in which a highly intelligence, powerful and economically useful system has non-complete preferences).
  The whole section at the end feels very confused to me. The author asserts that there is “an error” where people assert that “there are coherence theorems”, but man, that just seems like such a weird thing to argue for. Of course there are theorems that are relevant to the question of agent coherence, all of these seem really quite relevant. They might not prove the things in-practice, as many theorems tend to do.
  Like, I feel like with the same type of argument that is made in the post I could write a post saying “there are no voting impossibility theorems” and then go ahead and argue that the Arrow’s Impossibility Theorem assumptions are not universally proven, and then accuse everyone who ever talked about voting impossibility theorems that they are making “an error” since “those things are not real theorems”. And I think everyone working on voting-adjacent impossibility theorems would be pretty justifiedly annoyed by this.
  - EJT 21 Feb 2023 1:16 UTC
    21 points
    11 ∶ 1
    Parent
    I’m following previous authors in defining ‘coherence theorems’ as
    theorems which state that, unless an agent can be represented as maximizing expected utility, that agent is liable to pursue strategies that are dominated by some other available strategy.
    On that definition, there are no coherence theorems. VNM is not a coherence theorem, nor is Savage’s Theorem, nor is Bolker-Jeffrey, nor are Dutch Book Arguments, nor is Cox’s Theorem, nor is the Complete Class Theorem.
    there are theorems that are relevant to the question of agent coherence
    I’d have no problem with authors making that claim.
    I would urge the author to create an actual concrete situation that doesn’t seem very dumb in which a highly intelligence, powerful and economically useful system has non-complete preferences
    Working on it.
    - Habryka [Deactivated] 21 Feb 2023 4:12 UTC
      6 points
      2 ∶ 2
      Parent
      I’m following previous authors in defining ‘coherence theorems’ as
      Can you be concrete whose previous authors definition are you using here? A google search for your definition returns no results but this post, and this is definitely not a definition of “coherence theorems” that I would use.
      - EJT 21 Feb 2023 19:14 UTC
        14 points
        6 ∶ 0
        Parent
        Two points, made in order of importance:
        (1) How we define the term ‘coherence theorems’ doesn’t matter. What matters is that Premise 1 (striking out the word ‘coherence’, if you like) is false.
        (2) The way I define the term ‘coherence theorems’ seems standard.
        Now making point (1) in more detail:
        Reserve the term ‘coherence theorems’ for whatever you like. Premise 1 is false: there are no theorems which state that, unless an agent can be represented as maximizing expected utility, that agent is liable to pursue strategies that are dominated by some other available strategy. The VNM Theorem doesn’t say that, nor does Savage’s Theorem, nor does Bolker-Jeffrey, nor do Dutch Books, nor does Cox’s Theorem, nor does the Complete Class Theorem. That is the error in coherence arguments. Premise 1 is false.
        Now for point (2):
        I take the Appendix to make plausible enough that my use of the term ‘coherence theorems’ is standard, at least in online discussions. Here are some quotations.
        1.
        Now, by the general idea behind coherence theorems, since we can’t view this behavior as corresponding to expected utilities, we ought to be able to show that it corresponds to a dominated strategy somehow
        
        2.
        Roughly, the general claim these theorems make is that any system either (a) acts like an expected utility maximizer under some probabilistic model, or (b) throws away resources in a pareto-suboptimal manner.
        3.
        Summary: Violations of coherence constraints in probability theory and decision theory correspond to qualitatively destructive or dominated behaviors.
        Again, we see a manifestation of a powerful family of theorems showing that agents which cannot be seen as corresponding to any coherent probabilities and consistent utility function will exhibit qualitatively destructive behavior
        4.
        One of the most pleasing things about probability and expected utility theory is that there are many coherence arguments that suggest that these are the “correct” ways to reason. If you deviate from what the theory prescribes, then you must be executing a dominated strategy.
        5.
        ‘Coherence arguments’ mean that if you don’t maximize ‘expected utility’ (EU)—that is, if you don’t make every choice in accordance with what gets the highest average score, given consistent preferability scores that you assign to all outcomes—then you will make strictly worse choices by your own lights than if you followed some alternate EU-maximizing strategy (at least in some situations, though they may not arise). For instance, you’ll be vulnerable to ‘money-pumping’—being predictably parted from your money for nothing.
        6.
        The overall message here is that there is a set of qualitative behaviors and as long you do not engage in these qualitatively destructive behaviors, you will be behaving as if you have a utility function.
        7.
        I think that to contain the concept of Utility as it exists in me, you would have to do homework exercises I don’t know how to prescribe. Maybe one set of homework exercises like that would be showing you an agent, including a human, making some set of choices that allegedly couldn’t obey expected utility, and having you figure out how to pump money from that agent (or present it with money that it would pass up).
        8.
        The view that utility maximizers are inevitable is supported by a number of coherence theories developed early on in game theory which show that any agent without a consistent utility function is exploitable in some sense.
        Maybe the term ‘coherence theorems’ gets used differently elsewhere. That is okay. See point (1).
    - D0TheMath 21 Feb 2023 3:01 UTC
      0 points
      1 ∶ 1
      Parent
      
      Working on it.
      
      Spoiler (don’t read if you want to work on a fun puzzle or test your alignment metal).
      - Habryka [Deactivated] 21 Feb 2023 4:09 UTC
        7 points
        1 ∶ 0
        Parent
        Oh, nice, I do remember really liking that post. It’s a great example, though I think if you bring in time and trade-in-time back into this model you do actually get things that are more VNM-shaped again. But overall I am like “OK, I think that post actually characterizes how coherence arguments apply to agents without completeness quite well”, and am also like “yeah, and the coherence arguments still apply quite strongly, because they aren’t as fickle or as narrow as the OP makes them out to be”.
        But overall, yeah, I think this post would be a bunch stronger if it used the markets example from John’s post. I like it quite a bit, and I remember using it as an intuition pump in some situations that I somewhat embarrassingly failed to connect to this argument.
      - EJT 21 Feb 2023 3:08 UTC
        6 points
        2 ∶ 0
        Parent
        I cite John in the post!
        D0TheMath 21 Feb 2023 3:12 UTC
        1 point
        0 ∶ 0
        Parent
        Ah, ok. Why don’t you just respond with markets then!
  - Jan_Kulveit 21 Feb 2023 13:45 UTC
    10 points
    6 ∶ 0
    Parent
    You are correct with some of the criticism, but as a side-note, completeness is actually crazy.
    
    All real agents are bounded, and pay non-zero costs for bits, and as a consequence, don’t have complete preferences. Complete agents in real world do not exist. If they existed, correct intuitive model of them wouldn’t be ‘rational players’ but ‘utterly scary god, much bigger than the universe they live in’.
    - Habryka [Deactivated] 21 Feb 2023 19:26 UTC
      6 points
      1 ∶ 0
      Parent
      Oh, sorry, totally.
      
      The same is true for the other implicit assumption in VNM, which is doing bayesianism. There exist no bayesian agents. Any non-trivial bayesian agents would be similarly a terrifying alien god, much bigger than the universe they live in.
    - Jonas Moss 21 Feb 2023 14:28 UTC
      6 points
      2 ∶ 0
      Parent
      Do I understand you correctly here?
      
      Each agent has a computable partial preference ordering $x \leq y$ that decides if it prefers $x$ to $y$ .
      
      We’d like this partial relation to be complete (i.e., defined for all $x, y$ ) and transitive (i.e., $x \leq y$ and $y \leq z$ implies $x \leq z$ ).
      
      Now, if the relation is sufficiently non-trivial, it will be expensive to compute for some $x, y$ . So it’s better left undefined...?
      
      If so, I can surely relate to that, as I often struggle computing my preferences. Even if they are theoretically complete. But it seems to me the relationship is still defined, but might not be practical to compute.
      
      It’s also possible to think of it in this way: You start out with partial preference ordering, and need to calculate one of its transitive closures. But that is computationally difficult, and not unique either.
      
      I’m unsure what these observations add to the discussion, though.
  - David Johnston 21 Feb 2023 2:36 UTC
    10 points
    3 ∶ 0
    Parent
    and I would urge the author to create an actual concrete situation that doesn’t seem very dumb in which a highly intelligence, powerful and economically useful system has non-complete preferences
    I’d be surprised if you couldn’t come up with situations where completeness isn’t worth the cost—e.g. something like, to close some preference gaps you’d have to think for 100x as long, but if you close them all arbitrarily then you end up with intrasitivity.
    - Michael_PJ 21 Feb 2023 8:25 UTC
      6 points
      3 ∶ 0
      Parent
      This seems like a great point. Completeness requires closing all preference gaps, but if you do that inconsistently and violate transitivity then suddenly you are vulnerable to money-pumping.
  - NunoSempere 21 Feb 2023 2:38 UTC
    8 points
    2 ∶ 0
    Parent
    Like, I feel like with the same type of argument that is made in the post I could write a post saying “there are no voting impossibility theorems” and then go ahead and argue that the Arrow’s Impossibility Theorem assumptions are not universally proven, and then accuse everyone who ever talked about voting impossibility theorems that they are making “an error” since “those things are not real theorems”. And I think everyone working on voting-adjacent impossibility theorems would be pretty justifiedly annoyed by this.
    I think that there is some sense in which the character in your example would be right, since:
    Arrow’s theorem doesn’t bind approval voting.
    Generalizations of Arrow’s theorem don’t bind probabilistic results, e.g., each candidate is chosen with some probability corresponding to the amount of votes he gets.
    Like, if you had someone saying there was “a deep core of electoral process” which means that as they scale to important decisions means that you will necessarily get “highly defective electoral processes”, as illustrated in the classic example of the “dangers of the first pass the post system”. Well in that case it would be reasonable to wonder whether the assumptions of the theorem bind, or whether there is some system like approval voting which is much less shitty than the theorem provers were expecting, because the assumptions don’t hold.
    The analogy is imperfect, though, since approval voting is a known decent system, whereas for AI systems we don’t have an example friendly AI.
  - NunoSempere 21 Feb 2023 2:29 UTC
    7 points
    3 ∶ 0
    Parent
    (if this post is right)
    The post does actually seem wrong though.
    Glad that I added the caveat.
    Also, the title of “there are no coherence arguments” is just straightforwardly wrong. The theorems cited are of course real theorems, they are relevant to agents acting with a certain kind of coherence, and I don’t really understand the semantic argument that is happening where it’s trying to say that the cited theorems aren’t talking about “coherence”, when like, they clearly are.
    Well, part of the semantic nuance is that we don’t care as much about the coherence theorems that do exist if they will fail to apply to current and future machines
    IMO completeness seems quite reasonable to me and the argument here seems very weak (and I would urge the author to create an actual concrete situation that doesn’t seem very dumb in which a highly intelligence, powerful and economically useful system has non-complete preferences).
    Here are some scenarios:
    Our highly intelligent system notices that to have complete preferences over all trades would be too computationally expensive, and thus is willing to accept some, even a large degree of incompleteness.
    The highly intelligent system learns to mimic the values of human, which end up having non-complete preferences, which the agent mimics
    You train a powerful system to do some stuff, but also to detect when it is out of distribution and in that case do nothing. Assuming you can do that, their preference is incomplete, since when offered tradeoffs they always take the default option when out of distribution.
    The whole section at the end feels very confused to me. The author asserts that there is “an error” where people assert that “there are coherence theorems”, but man, that just seems like such a weird thing to argue for. Of course there are theorems that are relevant to the question of agent coherence, all of these seem really quite relevant. They might not prove the things in-practice, as many theorems tend to do.
    Mmh, then it would be good to differentiate between:
    There are coherence theorems that talk about some agents with some properties
    There are coherence theorems that prove that AI systems as will soon exist in the future will be optimizing utility functions
    You could also say a third thing, which would be: there are coherence theorems that strongly hint that AI systems as will soon exist in the future will be optimizing utility functions. They don’t prove it, but they make it highly probable because of such and such. In which case having more detail on the such and such would deflate most of the arguments in this post, for me.
    For instance:
    “‘Coherence arguments’ mean that if you don’t maximize ‘expected utility’ (EU)—that is, if you don’t make every choice in accordance with what gets the highest average score, given consistent preferability scores that you assign to all outcomes—then you will make strictly worse choices by your own lights than if you followed some alternate EU-maximizing strategy (at least in some situations, though they may not arise). For instance, you’ll be vulnerable to ‘money-pumping’—being predictably parted from your money for nothing.
    This is just false, because it is not taking into account the cost of doing expected value maximization, since giving consistent preferability scores is just very expensive and hard to do reliably. Like, when I poll people for their preferability scores, they give inconsistent estimates. Instead, they could be doing some expected utility maximization, but the evaluation steps are so expensive that I now basically don’t bother to do some more hardcore approximation of expected value for individuals, but for large projects and organizations. And even then, I’m still taking shortcuts and monkey-patches, and not doing pure expected value maximization.
    “This post gets somewhat technical and mathematical, but the point can be summarised as:
    You are vulnerable to money pumps only to the extent to which you deviate from the von Neumann-Morgenstern axioms of expected utility.
    In other words, using alternate decision theories is bad for your wealth.”
    The “in other words” doesn’t follow, since EV maximization can be more expensive than the shortcuts.
    Then there are other parts that give the strong impression that this expected value maximization will be binding in practice:
    “Rephrasing again: we have a wide variety of mathematical theorems all spotlighting, from different angles, the fact that a plan lacking in clumsiness, is possessing of coherence.”
    “The overall message here is that there is a set of qualitative behaviors and as long you do not engage in these qualitatively destructive behaviors, you will be behaving as if you have a utility function.”
    “The view that utility maximizers are inevitable is supported by a number of coherence theories developed early on in game theory which show that any agent without a consistent utility function is exploitable in some sense.”
    Here are some words I wrote that don’t quite sit right but which I thought I’d still share: Like, part of the MIRI beat as I understand it is to hold that there is some shining guiding light, some deep nature of intelligence that models will instantiate and make them highly dangerous. But it’s not clear to me whether you will in fact get models that instantiate that shining light. Like, you could imagine an alternative view of intelligence where it’s just useful monkey patches all the way down, and as we train more powerful models, they get more of the monkey patches, but without the fundamentals. The view in between would be that there are some monkey patches, and there are some deep generalizations, but then I want to know whether the coherence systems will bind to those kinds of agents.
    No need to respond/deeply engage, but I’d appreciate if you let me know if the above comments were too nitpicky.
  - D0TheMath 21 Feb 2023 3:02 UTC
    1 point
    0 ∶ 0
    Parent
    
    You can argue that the theorems are wrong, or that the explicit assumptions of the theorems don’t hold, which many people have done, but like, there are still coherence theorems, and IMO completeness seems quite reasonable to me and the argument here seems very weak (and I would urge the author to create an actual concrete situation that doesn’t seem very dumb in which a highly intelligence, powerful and economically useful system has non-complete preferences).
    
    If you want to see an example of this, I suggest John’s post here.
MichaelStJules 21 Feb 2023 5:02 UTC
23 points
4 ∶ 0
Related, but not so much the aim of your post: who or what is going to money pump or Dutch book a superintelligence even if the superintelligence doesn’t maximize expected utility? Many money pumps and Dutch books all seem pretty contrived and unlikely to occur naturally without adversaries. So, where would the pressure to avoid them actually come from? Maybe financial markets, but do they need to generalize their aversion to exploitation in markets to all their preferences? Negotiations with humans to gain power before it kills us all? Again, do they need to generalize from these?
I guess cyclic preferences could be bad in natural/non-adversarial situations.
There are also opposite pressures: if you would have otherwise had exploitable preferences, making them non-exploitable means giving something else up, i.e. some of your preference rankings. This is also a cost, and an AGI may not be willing to pay it.
- xuan 21 Feb 2023 13:25 UTC
  14 points
  5 ∶ 0
  Parent
  Seconded! On this note, I think the assumed presence of adversaries or competitors is actually one of the under-appreciated upshots of MIRI’s work on Logical Induction (https://intelligence.org/2016/09/12/new-paper-logical-induction/). By the logical induction criterion they propose, “good reasoning” is only defined with respect to a market of traders of a particular complexity class—which can be interpreted as saying that “good reasoning” is really intersubjective rather than objective! There’s only pressure to find the right logical beliefs in a reasonable amount of time if there are others who would fleece you for not doing so.
  - Nora Belrose 24 Mar 2023 9:12 UTC
    4 points
    2 ∶ 0
    Parent
    “good reasoning” is really intersubjective rather than objective! There’s only pressure to find the right logical beliefs in a reasonable amount of time if there are others who would fleece you for not doing so.
    This is a really interesting point that reminds me of arguments made by pragmatist philosophers like John Dewey and Richard Rorty. They also wanted to make “justification” an intersubjective phenomenon, of justifying your beliefs to other people. I don’t think they had money-pump arguments in mind though.
- Lauro Langosco 21 Feb 2023 14:44 UTC
  7 points
  1 ∶ 1
  Parent
  That’s why the standard prediction is not that AIs will be perfectly coherent, but that it makes sense to model them as being sufficiently coherent in practice, in the sense that e.g. we can’t rely on incoherence in order to shut them down.
- MichaelStJules 21 Feb 2023 18:03 UTC
  2 points
  0 ∶ 0
  Parent
  I guess there are acausal influence and locally multipolar (multiple competing AGI) cases, too.
David Johnston 21 Feb 2023 1:32 UTC
10 points
4 ∶ 0
I wonder if it is possible to derive expected utility maximisation type results from assumptions of “fitness” (as in, evolutionary fitness). This seems more relevant to the AI safety agenda—after all, we care about which kinds of AI are successful, not whether they can be said to be “rational”. It might also be a pathway to the kind of result AI safety people implicitly use—not that agents maximise some expected utility, but that they maximise utilities which force a good deal of instrumental convergence (i.e. describing them as expected utility maximisers is not just technically possible, but actually parsimonious). Actually, if we get the instrumental convergence then it doesn’t matter a great deal if the AIs aren’t strictly VNM rational.

In conclusion, I think we’re interested in results like fitness → instrumental convergence, not rationality → VNM utility.
I largely endorse the position that a number of AI safety people have seen theorems of the latter type and treated them as if that they imply theorems of the former type.
- Dan H 21 Feb 2023 4:28 UTC
  3 points
  1 ∶ 0
  Parent
  I agree fitness is a more useful concept than rationality (and more useful than an individual agent’s power), so here’s a document I wrote about it: https://drive.google.com/file/d/1p4ZAuEYHL_21tqstJOGsMiG4xaRBtVcj/view
Johan Gustafsson 21 Feb 2023 9:26 UTC
9 points
2 ∶ 1
It seems that your response to the money-pump argument is to give up decision-tree separability (and hence consequentialism). That amounts to a form of resolute choice, which i rebut in section 7.
What links here?
- EJT's comment on What’s Hard About The Shutdown Problem by johnswentworth (LessWrong; 31 Oct 2023 16:02 UTC; 6 points)
- EJT 22 Feb 2023 1:52 UTC
  4 points
  1 ∶ 0
  Parent
  Thanks for the comment! In this context, where we’re arguing about whether sufficiently-advanced artificial agents will satisfy the VNM axioms, I only have to give up Decision-Tree Separability*:
  Sufficiently-advanced artificial agents’ dispositions to choose options at a choice node will not depend on other parts of the decision tree than those that can be reached from that node.
  And Decision-Tree Separability* isn’t particularly plausible. It’s false if any sufficiently-advanced artificial agent acts in accordance with the following policy: ‘if I previously turned down some option X, I will not choose any option that I strictly disprefer to X.’ And it’s easy to see why agents might act in accordance with that policy: it makes them immune to money-pumps for Completeness.
  Also, it seems as if one of the major downsides of resolute choice is that agents sometimes have to act against their preferences. But, as I argue in the post, artificial agents with incomplete preferences who act in accordance with the policy above will never have to act against their preferences.
  - Johan Gustafsson 22 Feb 2023 3:54 UTC
    2 points
    0 ∶ 0
    Parent
    What you are suggesting is what I called “The Conservative Approach” to resolute choice, which I discuss critically on pages 73–74. It is not a new idea.
    Note also that avoiding money pumps for Completeness cannot alone motivate your suggested policy, since you can also avoid them by satisfying Completeness. So that argument does not work (without assuming the point at issue).
    Finally, I guess I don’t see why consequentialism would less plausible for artificial agents than other agents.
    - EJT 22 Feb 2023 23:38 UTC
      4 points
      1 ∶ 0
      Parent
      I didn’t mean to suggest it was new! I remember that part of your book.
      Your second point seems to me to get the dialectic wrong. We can read coherence arguments as saying:
      Sufficiently-advanced artificial agents won’t pursue dominated strategies, so they’ll have complete preferences.
      I’m pointing out that that inference is poor. Advanced artificial agents might instead avoid dominated strategies by acting in accordance with the policy that I suggest.
      I’m still thinking about your last point. Two quick thoughts:
      It seems like most humans aren’t consequentialists.
      Advanced artificial agents could have better memories of their past decisions than humans.
      - Johan Gustafsson 23 Feb 2023 0:40 UTC
        1 point
        0 ∶ 0
        Parent
        But my argument against proposals like yours is not that agents wouldn’t have sufficiently good memories. The objection (following Broome and others) is that the agents at node 2 have no reason at that node for ruling out option A- with your policy. The fact that A could have been chosen earlier should not concern you at node 2. A- is not dominated by any of the available options at node 2.
        Regarding the inference being poor, my argument in the book has two parts (1) the money pump for Completeness which relies on Decision-Tree Separability and (2) the defence of Decision-Tree Separability. It is (2) that rules out your proposal.
        Regarding your two quick thoughts, lots of people may be irrational. So that arguments does not work.
        EJT 23 Feb 2023 20:19 UTC
        7 points
        3 ∶ 0
        Parent
        I think all of these objections would be excellent if I were arguing against this claim:
        Agents are rationally required to satisfy the VNM axioms.
        But I’m arguing against this claim:
        Sufficiently-advanced artificial agents will satisfy the VNM axioms.
        And given that, I think your objections miss the mark.
        On your first point, I’m prepared to grant that agents have no reason to rule out option A- at node 2. All I need to claim is that advanced artificial agents might rule out option A- at node 2. And I think my argument makes that claim plausible.
        On your second point, Decision-Tree Separability doesn’t rule out my proposal. What would rule it out is Decision-Tree Separability*:
        sufficiently-advanced artificial agents’ dispositions to choose options at a choice node will not depend on other parts of the decision tree than those that can be reached from that node.
        And whatever the merits of Decision-Tree Separability, Decision-Tree Separability* seems to me not very plausible.
        On your third point, (whether or not most humans are irrational) most humans are non-consequentialists. So even if it is no more plausible that artificial agents will be non-consequentialists than humans, it can be plausible that artificial agents will be non-consequentialists. And it is relevant that advanced artificial agents could be better at remembering their past decisions than humans. That would make them better able to act in accordance with the policy that I suggest.
        What links here?
        The Shutdown Problem: Incomplete Preferences as a Solution by EJT (LessWrong; 23 Feb 2024 16:01 UTC; 53 points)
Shiney 21 Feb 2023 18:09 UTC
7 points
2 ∶ 0
I might have missed something but isn’t the “solution” to the concerns about the completeness money pump equivalent to the agent becoming complete.

E.g. after the agent has chose B over A it now effectively has a preference of B over A-.

I haven’t worked this through e.g. the proof of VNM etc. but are we sure this weaker notion of completeness might end up being enough to still get the relevant conclusions?

(quite busy might have a bit more of a think about this later)
What links here?
- EJT's comment on There are no coherence theorems by Dan H (LessWrong; 3 Jun 2023 19:05 UTC; 1 point)
- EJT 23 Feb 2023 0:30 UTC
  2 points
  0 ∶ 0
  Parent
  Nice point. The rough answer is ‘Yes, but only once the agent has turned down a sufficiently wide array of options.’ Depending on the details, that might never happen or only happen after a very long time.
  I’ve had a quick think about the more precise answer, and I think it is:
  - The agent’s preferences will be functionally complete once and only once it is the case that, for all pairs of options between which the agent has a preferential gap, the agent has turned down an option that is strictly preferred to one of the options in the pair.
  - Nick_Anyos 23 Feb 2023 4:29 UTC
    4 points
    1 ∶ 0
    Parent
    I had a similar thought to Shiny. Am I correct that an agent following your suggested policy (“‘if I previously turned down some option X, I will not choose any option that I strictly disprefer to X.’”) would never *appear* to violate completeness from the perspective of an observer that could only see their decisions and not their internal state? And assuming completeness is all we need to get to full utility maximization, does that mean an agent following your policy would act like a utility maximizer to an observer?
    - EJT 23 Feb 2023 19:35 UTC
      1 point
      0 ∶ 0
      Parent
      There’s a complication here related to a point that Rohin makes : if we can only see an agent’s decisions and we know nothing about its preferences, all behavior can be rationalized as EU maximization.
      But suppose we set that complication aside. Suppose we know this about an agent’s preferences:
      There is some option A such that the agent strictly prefers A+$1
      Then we can observe violations of Completeness. Suppose that we first offer our agent a choice between A and some other option B, and that the agent chooses A. Then we give the agent the chance to trade in A for B, and the agent takes the trade. That indicates that the agent does not strictly prefer A to B and does not strictly prefer B to A. Two possibilities remain: either the agent is indifferent between A and B, or the agent has a preferential gap between A and B.
      Now we offer our agent another choice: stick with B, or trade it in for A+$1. If the agent is indifferent between A and B, they will strictly prefer A+$1 to B (because indifference is sensitive to all sweetenings and sourings), and so we will observe the agent taking the trade. If we observe that the agent doesn’t take the trade, then they must have a preferential gap between A and B, and so their preferences must be incomplete.
finm 21 Feb 2023 10:50 UTC
4 points
2 ∶ 1
Looking forward to reading this. In the meantime, I notice that this post hasn’t been linked and seems likely to be relevant:
Coherence arguments do not entail goal-directed behavior by Rohin Shah
Jaime Sevilla 21 Feb 2023 5:05 UTC
3 points
0 ∶ 5
My quick take after skimming: I am quite confused about this post.
Of course the VNM theorem IS a coherence theorem.
How… could it not be a coherence theorem?
It tells you that actors following four intuitive properties can be represented as utility maximisers. We can quibble about the properties, but the result sounds important regardless for understanding agency!
The same reasoning could be applied to argue that Arrow’s Impossibility Theorem is Not Really About Voting. After all, we are just introducing all these assumptions about what good voting looks like!
- EJT 21 Feb 2023 5:39 UTC
  11 points
  6 ∶ 1
  Parent
  I would have hoped you reached the second sentence before skimming! I define what I mean (and what I take previous authors to mean) by ‘coherence theorems’ there.
  - David Johnston 21 Feb 2023 6:16 UTC
    11 points
    3 ∶ 0
    Parent
    I think your title might be causing some unnecessary consternation. “You don’t need to maximise utility to avoid domination” or something like that might have avoided a bit of confusion.
  - Jaime Sevilla 21 Feb 2023 6:14 UTC
    −5 points
    1 ∶ 5
    Parent
    Don’t get me wrong, I just think this is an extremely uncharitable and confusing way of presenting your work.
    I think it’s otherwise a great collection of coherence theorems and the discussion about completeness seems alright, though I haven’t read closely.
    - EJT 21 Feb 2023 7:22 UTC
      11 points
      7 ∶ 1
      Parent
      Using ‘coherence theorems’ with a meaning that is as standard as any, and explaining that meaning within two sentences, seems fine to me.
Jaime Sevilla 21 Feb 2023 4:52 UTC
2 points
0 ∶ 0
Not central to the argument, but I feel someone should be linking here to Garrabrant’s rejection of the independence axiom, which is fairly compelling IMO.