I think that in theory Max is right, that there’s some optimal way to have the best of both worlds. But in practice I think that there are pretty strong biases towards conformity, such that it’s probably worthwhile to shift the community as a whole indiscriminately towards being less epistemic modest.
As one example, people might think “I’ll make up my mind on small decisions, and defer on big decisions.” But then they’ll evaluate what feels big to them , rather than to the EA community overall, and thereby the community as a whole will end up being strongly correlated even on relatively small-scale bets. I think your comment itself actually makes this mistake—there’s now enough money in EA that, in my opinion, there should be many $5M grants which aren’t strongly correlated with the views of EA as a whole.
In particular, I note that venture capitalists allocate much larger amounts of money explicitly on anti-conformist principles. Maybe that’s because startups are a more heavy-tailed domain than altruism, and one where conformity is more harmful, but I’m not confident about that; the hypothesis that we just haven’t internalised the “hits-based” mentality as well as venture capitalists have also seems plausible.
(My best guess is that the average EA defers too much rather than too little. This and other comments on deference is to address specific points made, rather than to push any particular general takes).
Maybe that’s because startups are a more heavy-tailed domain than altruism, and one where conformity is more harmful
I think this is part of the reason. A plausibly bigger reason is that VC funding can’t result in heavy left-tails. Or rather, left-tails in VC funding are very rarely internalized. Concretely, if you pick your favorite example of “terrible startup for the future of sentient beings,” the VCs in question very rarely get in trouble, and approximately never get punished proportional to the counterfactual harm of their investments. VC funding can be negative for the VC beyond the opportunity cost of money (eg via reputational risk or whatever), but the punishment is quite low relative to the utility costs.
Obviously optimizing for increasing variance is a better deal when you clip the left tail, and optimizing for reducing variance is a better deal when you clip the right tail.
(I also independently think that heavy left tails in the utilitarian sense are probably less common in VC funding than in EA, but I think this is not necessary for my argument to go through).
I agree it’s possible that because of social pressures or similar things the best policy change that’s viable in practice could be an indiscriminate move toward more or less epistemic deference. Though I probably have less of a strong sense that that’s in fact true.
(Note that when implemented well, the “best of both worlds” policy could actually make it easier to express disagreement because it clarifies that there are two types of beliefs/credences to be kept track of separately, and that one of them has to exclude all epistemic deference.
Similarly, to the extent that people think that avoiding ‘bad, unilateral action’ is a key reason in favor of epistemic deference, it could actually “destigmatize” iconoclastic views if it’s common knowledge that an iconoclastic pre-deference view doesn’t imply unusual primarily-other-affecting actions because primarily-other-affecting actions depend on post-deference rather rather than pre-deference views.)
I agree with everything you say about $5M grants and VCs. I’m not sure if you think my mistake was mainly to consider a $5M stake a “large-scale” decision or something else, but if it’s the former I’m happy to concede that this wasn’t the best example to give for a decision where deference should get a lot of weight (though I think we agree that in theory it should get some weight?).
I think that in theory Max is right, that there’s some optimal way to have the best of both worlds. But in practice I think that there are pretty strong biases towards conformity, such that it’s probably worthwhile to shift the community as a whole indiscriminately towards being less epistemic modest.
As one example, people might think “I’ll make up my mind on small decisions, and defer on big decisions.” But then they’ll evaluate what feels big to them , rather than to the EA community overall, and thereby the community as a whole will end up being strongly correlated even on relatively small-scale bets. I think your comment itself actually makes this mistake—there’s now enough money in EA that, in my opinion, there should be many $5M grants which aren’t strongly correlated with the views of EA as a whole.
In particular, I note that venture capitalists allocate much larger amounts of money explicitly on anti-conformist principles. Maybe that’s because startups are a more heavy-tailed domain than altruism, and one where conformity is more harmful, but I’m not confident about that; the hypothesis that we just haven’t internalised the “hits-based” mentality as well as venture capitalists have also seems plausible.
(My best guess is that the average EA defers too much rather than too little. This and other comments on deference is to address specific points made, rather than to push any particular general takes).
I think this is part of the reason. A plausibly bigger reason is that VC funding can’t result in heavy left-tails. Or rather, left-tails in VC funding are very rarely internalized. Concretely, if you pick your favorite example of “terrible startup for the future of sentient beings,” the VCs in question very rarely get in trouble, and approximately never get punished proportional to the counterfactual harm of their investments. VC funding can be negative for the VC beyond the opportunity cost of money (eg via reputational risk or whatever), but the punishment is quite low relative to the utility costs.
Obviously optimizing for increasing variance is a better deal when you clip the left tail, and optimizing for reducing variance is a better deal when you clip the right tail.
(I also independently think that heavy left tails in the utilitarian sense are probably less common in VC funding than in EA, but I think this is not necessary for my argument to go through).
Good point, I agree this weakens my argument.
I agree it’s possible that because of social pressures or similar things the best policy change that’s viable in practice could be an indiscriminate move toward more or less epistemic deference. Though I probably have less of a strong sense that that’s in fact true.
(Note that when implemented well, the “best of both worlds” policy could actually make it easier to express disagreement because it clarifies that there are two types of beliefs/credences to be kept track of separately, and that one of them has to exclude all epistemic deference.
Similarly, to the extent that people think that avoiding ‘bad, unilateral action’ is a key reason in favor of epistemic deference, it could actually “destigmatize” iconoclastic views if it’s common knowledge that an iconoclastic pre-deference view doesn’t imply unusual primarily-other-affecting actions because primarily-other-affecting actions depend on post-deference rather rather than pre-deference views.)
I agree with everything you say about $5M grants and VCs. I’m not sure if you think my mistake was mainly to consider a $5M stake a “large-scale” decision or something else, but if it’s the former I’m happy to concede that this wasn’t the best example to give for a decision where deference should get a lot of weight (though I think we agree that in theory it should get some weight?).