lesswrong sequences volume 1.0

309
LessWrong.com Sequences C@Q@<I4L;BFNJBP Generated by lesswrong_book.py on 2013-01-05.  Pseudo-random version string: ee78df8a-6178-4d00-8a 06-16a9ed81a6be.

Upload: abhorsen205

Post on 02-Apr-2018

216 views

Category:

Documents


0 download

TRANSCRIPT

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 1/309

LessWrong.com Sequences

C@Q@<I4L;BFNJBP 

Generated by lesswrong_book.py on 2013-01-05. Pseudo-random version string: ee78df8a-6178-4d00-8a06-16a9ed81a6be.

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 2/309

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 3/309

 A collection of posts dealing with the fundamentals of  rationality: the difference between the map and the

territory, Bayes’s Theorem and the nature of evidence,why anyone should care about truth, and minds as

 reflective cognitive engines.

Part I

Map and Territory 

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 4/309

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 5/309

1. The Simple Truth 

`$I<D<D9<IK?@JG8G<I$NIFK<FE<O@JK<EK@8C@JD(P 

K<8:?<I>8M<@K98:BN@K?8E!.?<c;LE;<IC@E<;KIL<8E;KILK?N?<I<M<I@K8GG<8I<;@EK?<<JJ8PGIF989CP89FLKKN<EKPK@D<JN@K?8HL<JK@FED8IB9<J@;<<8:?.?<

 N8EK<;KFBEFNN?8K$D<8EK9PKILK?a g8E@<CC<>8EhAFLIE8C@JKi

 Author’s Foreword:

/?@J<JJ8P@JD<8EKKFI<JKFI<8E8@M<M@<NF=KILK?

.FD<FE< J8PJ KF PFL `(P D@I8:C< JE8B< F@C :8E I@; PFL F= CLE> :8E:<I @E ALJK K?I<< N<<BJa 4FL I<GCP `@;EcK 8 :C@E@:8C JKL;P J?FN K?@J :C8@D KF 9< LEKIL<a /?< FE< I<KLIEJ `/?@J EFK@FE F= bKILK?c @JHL@K<E8@M<N?8K;FPFLD<8E9PbKIL<ca

(8EP G<FGC< JF HL<JK@FE<; ;FEcK BEFN ?FN KF 8EJN<I @E<OHL@J@K<CP I@>FIFLJ ;<K8@C )FE<K?<C<JJ K?<P NFLC; EFK 9< N@J< KF898E;FE K?< :FE:<GK F= bKILK?c /?<I< N8J 8 K@D< N?<E EF FE< BE<N K?< <HL8K@FEJ F= >I8M@KP @E <OHL@J@K<CP I@>FIFLJ ;<K8@C P<K @= PFL

 N8CB<;F==8:C@==PFLNFLC;=8CC

*=K<E $ ?8M< J<<E f <JG<:@8CCP FE $EK<IE<K D8@C@E> C@JKJ f K?8K8D@;JK FK?<I :FEM<IJ8K@FE JFD<FE< J8PJ `3 @J KIL<a 8E; K?<E 8E8I>LD<EK 9I<8BJ FLK FM<I K?< LJ< F= K?< NFI; bKIL<c /?@J <JJ8P @J not D<8EK 8J 8E <E:P:CFG<;@: I<=<I<E:< =FI K?8K 8I>LD<EK -8K?<I $?FG< K?< 8I>L<IJ N@CC I<8; K?@J <JJ8P 8E; K?<E >F 98:B KF N?8K<M<IK?<P N<I< ;@J:LJJ@E> 9<=FI< JFD<FE< HL<JK@FE<; K?< E8KLI< F= KILK?

$E K?@J <JJ8P $ GFJ< HL<JK@FEJ $= PFL J<< N?8K J<<DJ C@B< 8 I<8CCP F9M@FLJ 8EJN<I @KcJ GIF989CP K?< 8EJN<I $ @EK<E; /?< F9M@eFLJ :?F@:< @JEcK always K?< 9<JK :?F@:< 9LK JFD<K@D<J 9P >FCCP @Kis $ ;FEcK JKFG CFFB@E> 8J JFFE $ =@E; 8E F9M@FLJ 8EJN<I 9LK @= $ >FFE CFFB@E> 8E; K?< F9M@FLJeJ<<D@E> 8EJN<I still  J<<DJ F9M@FLJ $;FEcK =<<C >L@CKP 89FLK B<<G@E> @K *? JLI< <M<IPFE< thinks KNF GCLJKNF @J =FLI <M<IPFE< says KNF GCLJ KNF @J =FLI 8E; @E K?< D<I< DLEe;8E< ;IL;><IP F= <M<IP;8P C@=< <M<IPFE< behaves 8J @= KNF GCLJ KNF@J =FLI 9LK N?8K ;F<J KNF GCLJ KNF really, ultimately <HL8C J E<8I8J $ :8E =@>LI< =FLI $KcJ JK@CC =FLI <M<E @= $ @EKFE< K?< HL<JK@FE @E 8 

http://yudkowsky.net/rational/the-simple-truth

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 6/309

JFC<DE GFIK<EKFLJ KFE< F= MF@:< /FF J@DGC< PFL J8P (8P9< FEK?@J F::8J@FE C@=< ;F<JEcK need KF 9< :FDGC@:8K<; 2FLC;EcK K?8K 9<I<=I<J?@E>

$= PFL 8I< FE< F= K?FJ< =FIKLE8K< =FCB KF N?FD K?< HL<JK@FEJ<<DJ KI@M@8C 8K K?< FLKJ<K $ ?FG< @K JK@CC J<<DJ KI@M@8C 8K K?< =@E@J?$= PFL =@E; PFLIJ<C= JKLDG<; 9P ;<<G 8E; D<8E@E>=LC HL<JK@FEJ I<eD<D9<I K?8K @= PFL BEFN <O8:KCP ?FN 8 JPJK<D NFIBJ 8E; :FLC;9L@C; FE< PFLIJ<C= FLK F= 9L:B<KJ 8E; G<99C<J @K J?FLC; EFK 9< 8 DPJK<IPKFPFL

$= :FE=LJ@FE K?I<8K<EJ N?<E PFL @EK<IGI<K 8 D<K8G?FI 8J 8 D<K8G?FIKIPK8B@E><M<IPK?@E> completely literally.

$D8>@E< K?8K @E 8E <I8 9<=FI< I<:FI;<; ?@JKFIP FI =FID8C D8K?e<D8K@:J $ 8D 8 J?<G?<I; 8E; $ ?8M< KIFL9C< KI8:B@E> DP J?<<G (P J?<<G JC<<G @E 8E <E:CFJLI< 8 =FC; 8E; K?< <E:CFJLI< @J ?@>? <EFL>?KF >L8I; DP J?<<G =IFD NFCM<J K?8K IF8D 9P E@>?K 8:? ;8P $ DLJKI<C<8J< DP J?<<G =IFD K?< =FC; KF G8JKLI< 8E; >I8Q< <8:? E@>?K $DLJK =@E; DP J?<<G 8E; I<KLIE K?<D KF K?< =FC; $= 8 J?<<G @J C<=KFLKJ@;< $ N@CC =@E; @KJ 9F;P K?< E<OK DFIE@E> B@CC<; 8E; ?8C=e<8Ke<E 9P NFCM<J LK @K @J JF ;@J:FLI8>@E> KF J:FLI K?< =@<C;J =FI ?FLIJCFFB@E> =FI FE< C8JK J?<<G N?<E $ BEFN K?8K GIF989CP 8CC K?< J?<<G

8I< @E K?< =FC; .FD<K@D<J $ >@M< LG <8ICP 8E; LJL8CCP $ ><K 8N8P  N@K? @K 9LK 8IFLE; 8 K<EK? F= K?< K@D< K?<I< @J 8 ;<8; J?<<G K?<E<OKDFIE@E>

$= FECP K?<I< N<I< JFD< N8P KF ;@M@E< N?<K?<I J?<<G 8I< JK@CC >I8Q@E> N@K?FLK K?< @E:FEM<E@<E:< F= CFFB@E> $ KIP J<M<I8C D<K?eF;J $ KFJJ K?< ;@M@E8K@FE JK@:BJ F= DP KI@9< $ KI8@E DP GJP:?@:GFN<IJ KF CF:8K< J?<<G K?IFL>? :C8@IMFP8E:< $ J<8I:? :8I<=LCCP =FII<8JFEJ KF 9<C@<M< 8CC K?< J?<<G 8I< @E K?< =FC; $K D8B<J EF ;@==<Ie<E:< IFLE; 8 K<EK? F= K?< K@D<J $ KLIE @E <8ICP $ =@E; 8 ;<8; J?<<G

K?< E<OK DFIE@E> +<I?8GJ $ I<8C@Q< K?8K DP D<K?F;J 8I<EcK NFIBe@E> 8E; G<I?8GJ $ :8I<=LCCP <O:LJ< <8:? =8@CLI< 9LK DP ;@C<DD8 @JJK@CC K?< J8D< $ :8E JG<E; 8E ?FLI J<8I:?@E> <M<IP GFJJ@9C< EFFB8E; :I8EEP N?<E DFJK F= K?< K@D< K?<I< 8I< EF I<D8@E@E> J?<<GFI $ :8E >F KF JC<<G <8ICP 8E; CFJ< FE K?< 8M<I8>< FE<eK<EK? F= 8 J?<<G

'8K< FE< 8=K<IEFFE $ =<<C <JG<:@8CCP K@I<; $ KFJJ K?< ;@M@E8K@FEJK@:BJ 8E; K?< ;@M@E8K@FE JK@:BJ J8P K?8K 8CC K?< J?<<G ?8M< I<KLIE<;

$ M@JL8C@Q< <8:? EFFB 8E; :I8EEP 8E; $ ;FEcK @D8>@E< J:IP@E> 8EP 

%(&,**!,'*1

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 7/309

J?<<G $cD JK@CC EFK :FE=@;<EK <EFL>? JF $ CFFB @EJ@;< K?< =FC; 8E;@K J<<DJ C@B< K?<I< 8I< 8 CFK F= J?<<G 8E; $ I<M@<N DP <8IC@<I <==FIKJ8E; ;<:@;< K?8K $ N8J <JG<:@8CCP ;@C@><EK /?@J ;@JJ@G8K<J DP 8EO@e

<KP 8E; $ >F KF JC<<G /?< E<OK DFIE@E> $ ;@J:FM<I two ;<8; J?<<G.FD<K?@E>@EJ@;<D<JE8GJ8E;$9<>@EK?@EB@E>:I<8K@M<CP

/?8K ;8P CFL; ?8DD<I@E> EF@J<J :FD< =IFD K?< >8K< F= K?<J?<<G=FC;cJ<E:CFJLI<

/?< E<OK DFIE@E> $ FG<E K?< >8K< F= K?< <E:CFJLI< FECP 8 C@KKC< N8P 8E; 8J <8:? J?<<G G8JJ<J FLK F= K?< <E:CFJLI< $ ;IFG 8 G<9e9C< @EKF 8 9L:B<K E8@C<; LG E<OK KF K?< ;FFI $E K?< 8=K<IEFFE 8J<8:? I<KLIE@E> J?<<G G8JJ<J 9P $ K8B< FE< G<99C< FLK F= K?< 9L:B<K2?<E K?<I< 8I< EF G<99C<J C<=K @E K?< 9L:B<K $ :8E JKFG J<8I:?@E> 

8E; KLIE @E =FI K?< E@>?K $K @J 8 brilliant EFK@FE $K N@CC I<MFCLK@FE@Q<J?<G?<I;@E>

/?8K N8J K?< K?<FIP $E GI8:K@:< @K KFFB :FEJ@;<I89C< I<=@E<eD<EK 9<=FI< K?< D<K?F; NFIB<; I<C@89CP .<M<I8C K@D<J $ J<8I:?<;=FI ?FLIJ 8E; ;@;EcK =@E; 8EP J?<<G 8E; K?< E<OK DFIE@E> K?<I<

 N<I< EF JKI8>>C<IJ *E <8:? F= K?<J< F::8J@FEJ @K I<HL@I<; ;<<GK?FL>?K KF =@>LI< FLK N?<I< DP 9L:B<K JPJK<D ?8; =8@C<; *E I<eKLIE@E> =IFD FE< =IL@KC<JJ J<8I:? $ K?FL>?K 98:B 8E; I<8C@Q<; K?8K

K?< 9L:B<K 8CI<8;P :FEK8@E<; G<99C<J N?<E $ JK8IK<; K?@J @K KLIE<;FLK N8J 8 98; @;<8 EFK?<I K@D< $ I8E;FDCP KFJJ<; G<99C<J @EKFK?< 9L:B<K KF 8DLJ< DPJ<C= 9<KN<<E K?< DFIE@E> 8E; K?< 8=K<IeEFFE K?@J KFF N8J 8 98; @;<8 8J $ I<8C@Q<; 8=K<I J<8I:?@E> =FI 8 =<N ?FLIJ LK $ GI8:K@:<; DP G<99C<:I8=K 8E; 9<:8D< 8 I<8JFE89CP GIF=@:@<EKG<99C<:I8=K<I

*E< 8=K<IEFFE 8 D8E I@:?CP 8KK@I<; @E N?@K< IF9<J C<8=P C8LI<CJJ8E;8CJ 8E; 9LJ@E<JJ JL@K KIL;><J @E 8CFE> K?< J8E;P KI8@C K?8K C<8;JKFDPG8JKLI<J

`8E$?<CGPFLa$@EHL@I</?< D8E K8B<J 8 98;>< =IFD ?@J :F8K 8E; =C@GJ @K FG<E GIFM@E> 

9<PFE; K?< J?8;FN F= 8 ;FL9K K?8K ?< @J (8IBFJ .FG?@JK@:LJ (8Oe@DLJ 8 ;<C<>8K< =IFD K?< .<E8K< F= -LD h*E< D@>?K NFE;<I

 N?<K?<I 8EFK?<I :FLC; JK<8C K?< 98;>< 9LK JF >I<8K @J K?< GFN<I F= K?<J< 98;><J K?8K @= 8EP FK?<I N<I< KF LJ< K?<D K?<P NFLC; @E K?8K@EJK8EK9< transformed @EKF(8IBFJi

,+!%($,*-,

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 8/309

`8CC D< (8IBa ?< J8PJ `$cD ?<I< KF :FE=@J:8K< K?< D8>@: G<9e9C<J @E K?< E8D< F= K?< .<E8K< 8IK@=8:KJ F= JL:? >I<8K GFN<I DLJKEFK=8CC@EKF@>EFI8EK?8E;Ja

`/?8K 9C<<;@Ec 8GGI<EK@:<a $ >IFLJ< LE;<I DP 9I<8K? `?<cJ 9<<E P8BB@Ec KF K?< M@CC8><IJ 8>8@Ea /?<E $ CFFB 8K (8IBcJ JK<IE =8:< 8E;J@>? `/?<P 8I<EcK D8>@: G<99C<Ja $ J8P 8CFL; `%LJK FI;@E8IP JKFE<J$G@:B<;LG=IFDK?<>IFLE;a

  =C@:B<I F= :FE=LJ@FE :IFJJ<J (8IBcJ =8:< K?<E ?< 9I@>?K<EJ8>8@E`$cD?<I<=FIK?<D8>@:9L:B<Ka?<;<:C8I<J

`$KcJ EFK 8 D8>@: 9L:B<Ka $ J8P N<8I@CP `$ LJ<; KF B<<G ;@IKP JF:BJ@E@Ka

(8IBcJ=8:<@JGLQQC<;`/?<EN?<I<@JK?<D8>@:a?<;<D8E;J E@EK<I<JK@E>HL<JK@FE`$KcJ?8I;KF<OGC8@Ea$J8P

(P :LII<EK 8GGI<EK@:< LKI<P 8KKI8:K<; 9P K?< :FDDFK@FE N8E;<IJ FM<I 8E; MFCLEK<<IJ ?@J <OGC8E8K@FE `$KcJ K?< C<M<C F= G<9e9C<J @E K?< 9L:B<Ka LKI<P J8PJ `/?<I<cJ 8 D8>@: C<M<C F= G<99C<J8E; PFL ?8M< KF ><K K?< C<M<C ALJK I@>?K FI @K ;F<JEcK NFIB $= PFLK?IFN @E DFI< G<99C<J FI K8B< JFD< FLK K?< 9L:B<K NFEcK 9< 8K K?<D8>@: C<M<C 8EPDFI< -@>?K EFN K?< D8>@: C<M<C @Ja LKI<P G<<IJ@EKFK?<9L:B<K`89FLKFE<eK?@I;=LCCa

`$ J<<a (8IB J8PJ <O:@K<;CP !IFD ?@J 98:B GF:B<K (8IB K8B<JFLK ?@J FNE 9L:B<K 8E; 8 ?<8G F= G<99C<J /?<E ?< >I89J 8 =<N ?8E;=LCJ F= G<99C<J 8E; JKL==J K?<D @EKF K?< 9L:B<K /?<E (8IBCFFBJ @EKF K?< 9L:B<K EFK@E> ?FN D8EP G<99C<J 8I< K?<I< `/?<I<

 N< >Fa (8IB J8PJ `K?< D8>@: C<M<C F= K?@J 9L:B<K @J ?8C= =LCC '@B<K?8Ka

`)Fa LKI<P J8PJ J?8IGCP `#8C= =LCC @J EFK K?< D8>@: C<M<C /?<D8>@: C<M<C @J 89FLK FE<eK?@I; #8C= =LCC @J ;<=@E@K<CP LED8>@: !LIe

K?<IDFI<PFLcI<LJ@E>K?<NIFE>9L:B<Ka(8IB KLIEJ KF D< GLQQC<; `$ K?FL>?K PFL J8@; K?< 9L:B<K N8JEcKD8>@:a

`$KcJ EFKa $ J8P J?<<G G8JJ<J FLK K?IFL>? K?< >8K< 8E; $ KFJJ8EFK?<I G<99C< @EKF K?< 9L:B<K `<J@;<J $cD N8K:?@E> K?< J?<<G/8CBKFLKI<Pa

(8IB ;L9@FLJCP <P<J K?< G<99C< $ KFJJ<; @E 9LK ;<:@;<J KF K<DeGFI8I@CP J?<CM< K?< HL<JK@FE (8IB KLIEJ KF LKI<P 8E; ;I8NJ ?@DeJ<C= LG ?8L>?K@CP `$KcJ 8 =I<< :FLEKIPa (8IB J8PJ `LE;<I K?< 9<E<Me

%(&,**!,'*1

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 9/309

FC<EK ;@:K8KFIJ?@G F= K?< .<E8K< F= :FLIJ< $ :8E ;IFG N?@:?<M<IG<99C<J$C@B<@EKFN?8K<M<I9L:B<K$C@B<a

 LKI<P :FEJ@;<IJ K?@J `)F PFL :8EcKa ?< J8PJ =@E8CCP `K?<I<

 NFEcK9<8EPD8>@:a`'FFBa J8PJ (8IB G8K@<EKCP `$ N8K:?<; PFL :8I<=LCCP 4FL

CFFB<; @E PFLI 9L:B<K :?<:B<; K?< C<M<C F= G<99C<J 8E; :8CC<; K?8KK?<D8>@:C<M<C$;@;<O8:KCPK?<J8D<K?@E>a

`/?8KcJEFK?FN@KNFIBJaJ8PJLKI<P

`*? $ J<<a J8PJ (8IB `$KcJ EFK K?< C<M<C F= G<99C<J @E my 9L:Be<K K?8KcJ D8>@: @KcJ K?< C<M<C F= G<99C<J @E your 9L:B<K $J K?8K N?8K

 PFL :C8@D 2?8K D8B<J PFLI 9L:B<K JF DL:? 9<KK<I K?8E D@E<

?L?a`2<CCa J8PJ LKI<P `@= N< N<I< KF <DGKP PFLI 9L:B<K 8E; K?<EGFLI 8CC K?< G<99C<J =IFD DP 9L:B<K @EKF PFLI 9L:B<K K?<E PFLI9L:B<K NFLC; ?8M< K?< D8>@: C<M<C /?<I<cJ 8CJF 8 GIF:<;LI< N< :8ELJ< KF :?<:B @= PFLI 9L:B<K ?8J K?< D8>@: C<M<C @= N< BEFN K?8K DP 9L:B<K ?8J K?< D8>@: C<M<C N< :8CC K?8K 8 9L:B<K :FDG8I< FG<I8eK@FEa

 EFK?<IJ?<<GG8JJ<J8E;$KFJJ@E8EFK?<IG<99C<

`#< ALJK KFJJ<; @E 8EFK?<I G<99C<a (8IB J8PJ `E; $ JLGGFJ<

 PFL :C8@D K?< E<N C<M<C @J 8CJF D8>@: $ :FLC; KFJJ G<99C<J @EKF PFLI9L:B<K LEK@C K?< C<M<C N8J K?< J8D< 8J D@E< 8E; K?<E FLI 9L:Be<KJ NFLC; 8>I<< 4FLcI< ALJK :FDG8I@E> DP 9L:B<K KF PFLI 9L:B<KKF ;<K<ID@E< N?<K?<I you K?@EB K?< C<M<C @J bD8>@:c FI EFK 2<CC $K?@EB your 9L:B<K @JEcK D8>@: 9<:8LJ< @K ;F<JEcK ?8M< K?< J8D< C<M<CF=G<99C<J8JD@E<.FK?<I<a

`28@KaJ8PJLKI<P`PFL;FEcKLE;<IJK8E;ea

`P bD8>@: C<M<Cc PFL D<8E J@DGCP K?< C<M<C F= G<99C<J @E PFLI

FNE 9L:B<K E; N?<E $ J8P bD8>@: C<M<Cc $ D<8E K?< C<M<C F= G<9e9C<J @E DP 9L:B<K /?LJ PFL CFFB 8K DP 9L:B<K 8E; J8P @K c@JEcKD8>@:c 9LK K?< NFI; bD8>@:c D<8EJ ;@==<I<EK K?@E>J KF ;@==<I<EKG<FGC< 4FL E<<; KF JG<:@=P whose D8>@: @K @J 4FL J?FLC; J8P K?8KDP 9L:B<K ;F<JEcK ?8M< cLKI<PcJ D8>@: C<M<Cc 8E; $ J8P K?8K PFLI9L:B<K ;F<JEcK ?8M< c(8IBcJ D8>@: C<M<Cc /?8K N8P K?< 8GG8I<EK:FEKI8;@:K@FE>F<J8N8Pa

`LKeaJ8PJLKI<P?<CGC<JJCP

,+!%($,*-,

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 10/309

`@==<I<EK G<FGC< :8E ?8M< ;@==<I<EK 9L:B<KJ N@K? ;@==<I<EK C<Me<CJ F= G<99C<J N?@:? GIFM<J K?@J 9LJ@E<JJ 89FLK bD8>@:c @J :FDGC<K<eCP8I9@KI8IP8E;JL9A<:K@M<a

`(8IBa$J8P`;@;8EPFE<K<CCPFLN?8KK?<J<G<99C<J do? a Do? aJ8PJ(8IB`$K?FL>?KK?<PN<I<ALJKD8>@:a

`$= K?< G<99C<J ;@;EcK ;F 8EPK?@E>a J8PJ LKI<P `FLI $.* GIF:<JJ <==@:@<E:P 8L;@KFI NFLC; <C@D@E8K< K?< GIF:<;LI< =IFD FLI;8@CPNFIBa

`2?8KcJPFLI8L;@KFIcJE8D<a

`8IN@EaJ8PJLKI<P

`#Da J8PJ (8IB `?8IC<J ;F<J ?8M< 8 I<GLK8K@FE 8J 8 JKI@:K 8Le

;@KFI .F ;F K?< G<99C<J 9C<JJ K?< =CF:BJ 8E; :8LJ< K?< @E:I<8J< F= J?<<Ga

`)Fa $ J8P `/?< M@IKL< F= K?< G<99C<J @J K?@J @= N< CFFB @EKF K?<9L:B<K 8E; J<< K?< 9L:B<K @J <DGKP F= G<99C<J N< BEFN K?< G8JeKLI<J 8I< C@B<N@J< <DGKP F= J?<<G $= N< ;F EFK LJ< K?< 9L:B<K N<DLJK J<8I:? 8E; J<8I:? LEK@C ;8IB C<JK FE< C8JK J?<<G I<D8@E *I @= 

 N< JKFG FLI NFIB <8ICP K?<E JFD<K@D<J K?< E<OK DFIE@E> N< =@E; 8 ;<8; J?<<G =FI K?< NFCM<J J8M8>< 8EP J?<<G C<=K FLKJ@;< $= N< CFFB@E K?< 9L:B<K N< BEFN N?<E 8CC K?< J?<<G 8I< ?FD< 8E; N< :8E

I<K@I<N@K?FLK=<8Ia(8IB :FEJ@;<IJ K?@J `/?8K JFLE;J I8K?<I @DGC8LJ@9C<a ?< J8PJ

<M<EKL8CCP `@; PFL :FEJ@;<I LJ@E> ;@M@E8K@FE JK@:BJ @M@E8K@FEJK@:BJ 8I< @E=8CC@9C< FI 8K C<8JK 8EPFE< N?F J8PJ K?<P 8I< =8CC@9C< @J9LIE<; 8K K?< JK8B< /?@J @J 8E <OKI<D<CP G8@E=LC N8P KF ;@< @K =FCeCFNJK?8K;@M@E8K@FEJK@:BJ8I<@E=8CC@9C<a

`4FLcI<N<C:FD<KFLJ<;@M@E8K@FEJK@:BJ@=PFLC@B<a$J8P

`*? >FF; ?<8M<EJ F= :FLIJ< EFKa J8PJ (8IB `/?<P NFIB @Ee

=8CC@9CP N@K? 89JFCLK< G<I=<:K@FE FE <M<IP F::8J@FE 8J 9<=@KJ JL:?9C<JJ<; @EJKILD<EKJ 9LK N?8K @= K?<I< N<I< 8 ;<8; J?<<G K?< E<OKDFIE@E> $ FECP LJ< K?< ;@M@E8K@FE JK@:BJ N?<E K?<I< @J EF GFJJ@9@Ce@KP F= K?<@I 9<@E> GIFM<E NIFE> *K?<IN@J< $ D@>?K 9< 9LIE<; 8C@M<.F?FN;F<JPFLID8>@:9L:B<KNFIBa

#FN ;F<J K?< 9L:B<K NFIB_ $c; 9<KK<I JK8IK N@K? K?< J@DGC<JKGFJJ@9C< :8J< `2<CCa $ J8P `JLGGFJ< K?< G8JKLI<J 8I< <DGKP 8E;K?< 9L:B<K @JEcK <DGKP /?<E N<cCC N8JK< ?FLIJ CFFB@E> =FI 8 J?<<GK?8K @JEcK K?<I< E; @= K?<I< 8I< J?<<G @E K?< G8JKLI<J 9LK K?< 9L:Be<K @J <DGKP K?<E LKI<P 8E; $ N@CC KLIE @E KFF <8ICP 8E; N<cCC =@E;

%(&,**!,'*1

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 11/309

;<8; J?<<G K?< E<OK DFIE@E> .F 8E <DGKP 9L:B<K @J D8>@:8C @= 8E;FECP@=K?<G8JKLI<J8I<<DGKPea

`#FC; FEa J8PJ LKI<P `/?8K JFLE;J C@B< 8 M8:LFLJ K8LKFCF>P 

KF D< I<EcK 8E <DGKP 9L:B<K 8E; <DGKP G8JKLI<J F9M@FLJCP K?<J8D<K?@E>a

`$KcJ EFK M8:LFLJa $ J8P `#<I<cJ 8E 8E8CF>P /?< CF>@:@8E C=I<;/8IJB@ FE:< J8@; K?8K K?< 8JJ<IK@FE b.EFN @J N?@K<c @J KIL< @= 8E; FECP @= JEFN @J N?@K< $= PFL :8E LE;<IJK8E; K?8K PFL J?FLC; 9< 89C< KFJ<< N?P 8E <DGKP 9L:B<K @J D8>@:8C @= 8E; FECP @= K?< G8JKLI<J 8I<<DGKPF=J?<<Ga

`#FC; FEa J8PJ (8IB `/?<J< 8I< buckets /?<P ;FEcK ?8M< 8EPeK?@E> KF ;F N@K? sheep L:B<KJ 8E; J?<<G 8I< F9M@FLJCP :FDGC<K<CP ;@==<I<EK /?<I<cJ EF N8P K?< J?<<G :8E <M<I @EK<I8:K N@K? K?< 9L:Be<Ka

`/?<E N?<I< ;F you K?@EB K?< D8>@: :FD<J =IFDa @EHL@I<J LKI<P

(8IB :FEJ@;<IJ `4FL J8@; PFL :FLC; :FDG8I< KNF 9L:B<KJ KF:?<:B @= K?<P ?8; K?< J8D< C<M<C_ $ :8E J<< ?FN 9L:B<KJ :8E @EK<Ie8:K N@K? 9L:B<KJ (8P9< N?<E PFL ><K 8 C8I>< :FCC<:K@FE F= 9L:B<KJ8E; K?<P 8CC ?8M< K?< J8D< C<M<C that’s N?8K ><E<I8K<J K?< D8>@: $cCC

:8CCK?8KK?<:F?<I<EK@JKK?<FIPF=D8>@:9L:B<KJa`$EK<I<JK@E>a J8PJ LKI<P `$ BEFN K?8K DP D8JK<I @J NFIB@E> 

FE 8 JPJK<D N@K? DLCK@GC< 9L:B<KJ f ?< J8PJ @K D@>?K NFIB 9<KK<I9<:8LJ< F= bI<;LE;8E:Pc 8E; b<IIFI :FII<:K@FEc /?8K JFLE;J C@B< :Fe?<I<EK@JDKFD<a

`/?<PcI<EFKHL@K<K?<J8D<ea$JK8IKKFJ8P

`'<KcJ K<JK K?< :F?<I<EK@JD K?<FIP F= D8>@:a J8PJ LKI<P `$ :8EJ<< PFLcM< >FK =@M< DFI< 9L:B<KJ @E PFLI 98:B GF:B<K $cCC ?8E; PFL

K?< 9L:B<K N<cI< LJ@E> 8E; K?<E PFL :8E =@CC LG PFLI FK?<I 9L:B<KJKFK?<J8D<C<M<Cea

(8IB I<:F@CJ @E ?FIIFI `.KFG /?<J< 9L:B<KJ ?8M< 9<<E G8JJ<;;FNE @E DP =8D@CP =FI ><E<I8K@FEJ 8E; K?<PcM< 8CN8PJ ?8; K?< J8D<C<M<C $= $ 8::<GK PFLI 9L:B<K DP 9L:B<K :FCC<:K@FE N@CC 9<:FD< C<JJ:F?<I<EK8E;K?<D8>@:N@CC>F8N8Pa

`LK PFLI current  9L:B<KJ ;FEcK ?8M< 8EPK?@E> KF ;F N@K? K?<J?<<GaGIFK<JKJLKI<P

,+!%($,*-,

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 12/309

(8IB CFFBJ <O8JG<I8K<; `'FFB $cM< <OGC8@E<; 9<=FI< K?<I<cJF9M@FLJCP EF N8P K?8K J?<<G :8E @EK<I8:K N@K? 9L:B<KJ L:B<KJ :8EFECP@EK<I8:KN@K?FK?<I9L:B<KJa

`$KFJJ@E8G<99C<N?<E<M<I8J?<<GG8JJ<Ja$GF@EKFLK`2?<E 8 J?<<G G8JJ<J PFL KFJJ @E 8 G<99C<a (8IB J8PJ `2?8K

;F<JK?8K?8M<KF;FN@K?8EPK?@E>a

`$KcJ 8E @EK<I8:K@FE 9<KN<<E K?< J?<<G 8E; K?< G<99C<Ja $ I<GCP

`)F @KcJ 8E @EK<I8:K@FE 9<KN<<E K?< G<99C<J 8E; youa (8IBJ8PJ `/?< D8>@: ;F<JEcK :FD< =IFD K?< J?<<G @K :FD<J =IFD you(<I< J?<<G 8I< F9M@FLJCP EFED8>@:8C /?< D8>@: ?8J KF :FD< =IFD somewhereFEK?<N8PKFK?<9L:B<Ka

$ GF@EK 8K 8 NFF;<E D<:?8E@JD G<I:?<; FE K?< >8K< `F PFLJ<< K?8K =C8G F= :CFK? ?8E>@E> ;FNE =IFD K?8K NFF;<E :FEKI8GK@FE2<cI< JK@CC =@;;C@E> N@K? K?8K f @K ;F<JEcK NFIB I<C@89CP f 9LK N?<EJ?<<G G8JJ K?IFL>? K?<P ;@JKLI9 K?< :CFK? 2?<E K?< :CFK? DFM<J8J@;< 8 G<99C< ;IFGJ FLK F= 8 I<J<IMF@I 8E; =8CCJ @EKF K?< 9L:B<K/?8K N8P LKI<P 8E; $ NFEcK ?8M< KF KFJJ @E K?< G<99C<J FLIJ<CM<Ja

(8IB =LIIFNJ ?@J 9IFN `$ ;FEcK HL@K< =FCCFN PFL_ @J K?< clothD8>@:8Ca

$ J?IL> `$ FI;<I<; @K FEC@E< =IFD 8 :FDG8EP :8CC<; )8KLI8C .<e

C<:K@FEJ /?< =89I@: @J :8CC<; .<EJFIP (F;8C@KPa $ G8LJ< J<<@E> K?<@E:I<;LCFLJ <OGI<JJ@FEJ F= (8IB 8E; LKI<P `$ 8;D@K K?< E8D<J8I< 8 9@K )<N ><P /?< GF@EK @J K?8K 8 G8JJ@E> J?<<G KI@>><IJ 8 :?8@E F= :8LJ< 8E; <==<:K K?8K <E;J N@K? 8 G<99C< @E K?< 9L:B<K Af-terward PFL:8E:FDG8I<K?<9L:B<KKFFK?<I9L:B<KJ8E;JFFEa

`$ JK@CC ;FEcK ><K @Ka (8IB J8PJ `4FL :8EcK =@K 8 J?<<G @EKF 8 9L:B<K *ECP G<99C<J >F @E 9L:B<KJ 8E; @KcJ F9M@FLJ K?8K G<99C<JFECP@EK<I8:KN@K?FK?<IG<99C<Ja

`/?< J?<<G @EK<I8:K N@K? K?@E>J K?8K @EK<I8:K N@K? G<99C<J_a $J<8I:? =FI 8E 8E8CF>P `.LGGFJ< PFL CFFB ;FNE 8K PFLI J?F<C8:<J  G?FKFE C<8M<J K?< .LE K?<E KI8M<CJ ;FNE K?IFL>? 8IK?cJ 8KeDFJG?<I< K?<E 9FLE:<J F== PFLI J?F<C8:<J K?<E G8JJ<J K?IFL>?K?< GLG@C F= PFLI <P< K?<E JKI@B<J K?< I<K@E8 K?<E @J 89JFI9<; 9P 8 IF; FI 8 :FE< /?< G?FKFEcJ <E<I>P D8B<J K?< 8KK8:?<; E<LIFE=@I< N?@:? :8LJ<J FK?<I E<LIFEJ KF =@I< E<LI8C 8:K@M8K@FE G8KK<IE@E PFLI M@JL8C :FIK<O :8E @EK<I8:K N@K? PFLI 9<C@<=J 89FLK PFLIJ?F<C8:<J J@E:< 9<C@<=J 89FLK J?F<C8:<J 8CJF <O@JK @E E<LI8C JL9e

%(&,**!,'*1

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 13/309

JKI8K< $= PFL :8E LE;<IJK8E; K?8K PFL J?FLC; 9< 89C< KF J<< ?FN 8 G8JJ@E>J?<<G:8LJ<J8G<99C<KF<EK<IK?<9L:B<Ka

`K <O8:KCP which GF@EK @E K?< GIF:<JJ ;F<J K?< G<99C< 9<:FD<

D8>@:aJ8PJ(8IB`$K_ LD_a )FN I’m JK8IK@E> KF ><K :FE=LJ<; $ J?8B< DP ?<8;

KF :C<8I 8N8P :F9N<9J /?@J 8CC J<<D<; J@DGC< <EFL>? N?<E $ NFB<LG K?@J DFIE@E> 8E; K?< G<99C<e8E;e9L:B<K JPJK<D ?8JEcK >FKK<E8EP DFI< :FDGC@:8K<; J@E:< K?<E `/?@J @J 8 CFK <8J@<I KF LE;<IJK8E;@= PFL I<D<D9<I K?8K K?< point  F= K?< JPJK<D @J KF B<<G KI8:B F= J?<<Ga

(8IB J@>?J J8;CP `)<M<I D@E;_ @KcJ F9M@FLJ PFL ;FEcK BEFN(8P9< 8CC G<99C<J 8I< D8>@:8C KF JK8IK N@K? <M<E 9<=FI< K?<P <EK<IK?<9L:B<K2<:FLC;:8CCK?8KGFJ@K@FEG8EG<99C@JDa

`#8a LKI<P J8PJ J:FIE I@:? @E ?@J MF@:< `(<I< N@J?=LC K?@EBe@E> )FK 8CC G<99C<J 8I< :I<8K<; <HL8C /?< G<99C<J @E your 9L:B<K8I< not D8>@:8C/?<PcI<FECPCLDGJF=JKFE<a

(8IBcJ =8:< KLIEJ JK<IE `)FNa ?< :I@<J `EFN PFL J<< K?< ;8Ee ><I F= K?< IF8; PFL N8CB *E:< PFL J8P K?8K JFD< G<FGC<cJ G<99C<J8I< D8>@:8C 8E; JFD< 8I< EFK PFLI GI@;< N@CC :FEJLD< PFL 4FL N@CCK?@EB PFLIJ<C= JLG<I@FI KF 8CC FK?<IJ 8E; JF =8CC (8EP K?IFL>?FLK

?@JKFIP ?8M< KFIKLI<; 8E; DLI;<I<; 9<:8LJ< K?<P K?FL>?K K?<@I FNEG<99C<J JLGI<D<a K@E>< F= :FE;<J:<EJ@FE <EK<IJ (8IBcJ MF@:<`2FIJ?@GG@E> 8 C<M<C F= G<99C<J 8J bD8>@:8Cc @DGC@<J K?8K K?<I<cJ 8E89JFCLK< G<99C< C<M<C @E 8 .LGI<D< L:B<K )F9F;P 9<C@<M<J @E 8 .LGI<D<L:B<KK?<J<;8PJa

`*E<a $ J8P `.?<<G 8I< EFK 89JFCLK< G<99C<J /NF $ ;FEcKK?@EB DP 9L:B<K 8:KL8CCP :FEK8@EJ K?< J?<<G /?I<< $ ;FEcK NFIJ?@GDP 9L:B<K C<M<C 8J G<I=<:K f $ 8;ALJK @K JFD<K@D<J f 8E; $ ;F K?8Kbecause $:8I<89FLKK?<J?<<Ga

`<J@;<Ja J8PJ LKI<P `JFD<FE< N?F 9<C@<M<J K?8K GFJJ<JJ@E> 89JFCLK< G<99C<J would C@:<EJ< KFIKLI< 8E; DLI;<I @J D8B@E> 8 D@JeK8B< K?8K ?8J EFK?@E> KF ;F N@K? 9L:B<KJ 4FLcI< JFCM@E> K?< NIFE> GIF9C<Da

(8IB :8CDJ ?@DJ<C= ;FNE `$ JLGGFJ< $ :8EcK <OG<:K 8EP 9<KK<I=IFD D<I< J?<G?<I;J 4FL GIF989CP 9<C@<M< K?8K JEFN @J N?@K<;FEcKPFLa

`0D_P<JaJ8PJLKI<P

,+!%($,*-,

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 14/309

`$K ;F<JEcK 9FK?<I PFL K?8K Joseph Stalin 9<C@<M<; K?8K JEFN @J N?@K<a

`0D_EFaJ8PJLKI<P

(8IB >8Q<J @E:I<;LCFLJCP 8K LKI<P 8E; =@E8CCP J?IL>J `'<KcJJLGGFJ< GLI<CP =FI K?< J8B< F= 8I>LD<EK K?8K PFLI G<99C<J 8I<D8>@:8C8E;D@E<8I<EcK8EPFLK<CCD<N?8KK?<;@==<I<E:<@Ja

`(P G<99C<J represent  K?< J?<<Ga LKI<P J8PJ KI@LDG?8EKCP`Your  G<99C<J ;FEcK ?8M< K?< I<GI<J<EK8K@M<E<JJ GIFG<IKP JF K?<P 

 NFEcK NFIB /?<P 8I< <DGKP F= D<8E@E> %LJK CFFB 8K K?<D /?<I<cJEF 8LI8 F= J<D8EK@: :FEK<EK K?<P 8I< D<I<CP G<99C<J 4FL E<<; 8 9L:B<KN@K?JG<:@8C:8LJ8CGFN<IJa

`?a(8IBJ8PJ`.G<:@8C:8LJ8CGFN<IJ@EJK<8;F=D8>@:a`O8:KCPa J8PJ LKI<P `$cD EFK JLG<IJK@K@FLJ +FJKLC8K@E> D8>e@: @E K?@J ;8P 8E; 8>< NFLC; 9< LE8::<GK89C< KF K?< @EK<IE8K@FE8CJ?<G?<I;@E> :FDDLE@KP 2< ?8M< =FLE; K?8K GFJKLC8K@E> D8>@:J@DGCP ;F<JEcK NFIB 8J 8E <OGC8E8K@FE =FI J?<G?<I;@E> G?<EFD<E8.F N?<E $ J<< JFD<K?@E> $ ;FEcK LE;<IJK8E; 8E; $ N8EK KF <OGC8@E@K LJ@E> 8 DF;<C N@K? EF @EK<IE8C ;<K8@C K?8K D8B<J EF GI<;@:K@FEJ<M<E @E I<KIFJG<:K $ GFJKLC8K< JG<:@8C :8LJ8C GFN<IJ $= K?8K ;F<JEcK

 NFIB$cCCDFM<FEKF:8CC@E>@K8E<D<I><EKG?<EFD<EFEa

`2?8K B@E; F= JG<:@8C GFN<IJ ;F<J K?< 9L:B<K ?8M<a 8JBJ (8IB`#Da J8PJ LKI<P `(8P9< K?@J 9L:B<K @J @D9L<; N@K? 8E about-

 ness I<C8K@FE KF K?< G8JKLI<J /?8K NFLC; <OGC8@E N?P @K NFIB<; f  N?<EK?<9L:B<K@J<DGKP@K means K?<G8JKLI<J8I<<DGKPa

`2?<I< ;@; PFL =@E; K?@J 9L:B<Ka J8PJ (8IB `E; ?FN ;@; PFLI<8C@Q<@K?8;8E89FLKeE<JJI<C8K@FEKFK?<G8JKLI<Ja

`$KcJ 8E ordinary bucket a $ J8P `$ LJ<; KF :C@D9 KI<<J N@K? @K_ $;FEcKK?@EBK?@JHL<JK@FE needs KF9<;@==@:LCKa

`$cDK8CB@E>KFLKI<PaJ8PJ(8IB`4FL ?8M< KF 9@E; K?< 9L:B<K KF K?< G8JKLI<J 8E; K?< G<99C<J

KF K?< J?<<G LJ@E> 8 D8>@:8C I@KL8C f G8I;FE D< 8E <D<I><EKGIF:<JJ N@K? JG<:@8C :8LJ8C GFN<IJ f K?8K DP D8JK<I ;@J:FM<I<;a

 LKI<P<OGC8@EJ

 LKI<P K?<E 8KK<DGKJ KF ;<J:I@9< K?< I@KL8C N@K? (8IB EF;;@E> 8CFE>@EJ8><:FDGI<?<EJ@FE

%(&,**!,'*1

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 15/309

`4FL ?8M< KF K?IFN @E 8 G<99C< every K@D< 8 J?<<G C<8M<JK?IFL>? K?< >8K<a J8PJ (8IB `/8B< FLK 8 G<99C< every K@D< 8 J?<<GI<KLIEJa

 LKI<PEF;J`4<8?a`/?8KDLJK9<I<8CCP?8I;a(8IBJ8PJJPDG8K?<K@:8CCP

 LKI<P 9I@>?K<EJ JF8B@E> LG (8IBcJ JPDG8K?P C@B< I8@E `O8:KeCPa J8PJ LKI<P `$KcJ extremely ?8I; FE PFLI <DFK@FEJ 2?<E K?<9L:B<K ?8J ?<C; @KJ C<M<C =FI 8 N?@C< PFL_ K<E; KF ><K 8KK8:?<; KFK?8KC<M<Ca

  J?<<G G8JJ<J K?<E C<8M@E> K?IFL>? K?< >8K< LKI<P J<<J ?<JKFFGJ G@:BJ LG 8 G<99C< ?FC;J @K 8CF=K @E K?< 8@I `<?FC;a LKI<P 

GIF:C8@DJ ` J?<<G ?8J G8JJ<; $ DLJK EFN KFJJ 8 G<99C< @EKF K?@J9L:B<K DP ;<8I 9L:B<K 8E; ;<JKIFP K?8K =FE; C<M<C N?@:? ?8J ?<C;=FI JF CFE> f a EFK?<I J?<<G G8JJ<J LKI<P :8L>?K LG @E ?@J ;I8eD8 D@JJ<J @K JF $ GCLEB 8 G<99C< @EKF K?< 9L:B<K LKI<P @J JK@CCJG<8B@E> a f =FI K?8K @J K?< JLGI<D< K<JK F= K?< J?<G?<I; KF K?IFN @E K?< G<99C< 9< @K <M<I JF 8>FE@Q@E> 9< K?< FC; C<M<C <M<I JF GI<e:@FLJ $E;<<; FECP K?< 9<JK F= J?<G?<I;J :8E D<<K 8 I<HL@I<D<EKJFJK<IEe`

`LKI<Pa $ J8P `@= PFL N8EK KF 9< 8 >I<8K J?<G?<I; JFD<;8P

C<8IE KF J?LK LG 8E; K?IFN @E K?< G<99C< )F =LJJ )F ;I8D8 %LJK;F@Ka

`E; K?@J I@KL8Ca J8PJ (8IB `@K 9@E;J K?< G<99C<J KF K?< J?<<G9P K?< D8>@:8C C8NJ F= .PDG8K?P 8E; FEK8>@FE C@B< 8 MFF;FF ;FCCa

 LKI<P N@E:<J 8E; CFFBJ 8IFLE; `+C<8J< FEcK :8CC @K .PDG8K?P 8E; FEK8>@FE 2< J?<G?<I;J 8I< 8E 8EK@eJLG<IJK@K@FLJ =FCB 0J<K?<NFI;b@EK<EK@FE8C@KPcFIJFD<K?@E>C@B<K?8Ka

`8E$CFFB8K8G<99C<aJ8PJ(8IB

`.LI<a $ J8P $ K8B< FE< F= K?< G<99C<J FLK F= K?< 9L:B<K 8E;KFJJ @K KF (8IB /?<E $ I<8:? KF K?< >IFLE; G@:B LG 8EFK?<I G<9e9C<8E;;IFG@K@EKFK?<9L:B<K

 LKI<PCFFBJ8KD<GLQQC<;`@;EcKPFLALJKD<JJ@KLGa

$ J?IL> `$ ;FEcK K?@EB JF 2<cCC BEFN $ D<JJ<; @K LG @= K?<I<cJ 8 ;<8; J?<<G E<OK DFIE@E> FI @= N< J<8I:? =FI 8 =<N ?FLIJ 8E; ;FEcK=@E;8EPJ?<<Ga

`LKeaLKI<PJ8PJ

,+!%($,*-,

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 16/309

`$ K8L>?K PFL <M<IPK?@E> you BEFN 9LK $ ?8M<EcK K8L>?K PFL <Me<IPK?@E> I BEFNa$J8P

(8IB @J <O8D@E@E> K?< G<99C< JK8I@E> 8K @K @EK<EKCP #< ?FC;J

?@J ?8E; FM<I K?< G<99C< 8E; DLKK<IJ 8 =<N NFI;J K?<E J?8B<J ?@J?<8; `$ ;FEcK J<EJ< 8EP D8>@:8C GFN<Ia ?< J8PJ `+8I;FE D< $;FEcKJ<EJ<8EP@EK<EK@FE8C@KPa

` G<99C< FECP ?8J @EK<EK@FE8C@KP @= @KcJ @EJ@;< 8 D8e 8E <D<I><EK9L:B<KaJ8PJLKI<P`*K?<IN@J<@KcJALJK8D<I<G<99C<a

`)FK 8 GIF9C<Da $ J8P $ K8B< 8 G<99C< FLK F= K?< 9L:B<K 8E;KFJJ @K 8N8P /?<E $ N8CB FM<I KF N?<I< (8IB JK8E;J K8G ?@J ?8E;?FC;@E> 8 G<99C< 8E; J8P `$ ;<:C8I< K?@J ?8E; KF 9< G8IK F= K?< D8>e@:9L:B<Ka/?<E$I<JLD<DPGFJK8KK?<>8K<J

 LKI<PC8L>?J`)FNPFLcI<ALJK9<@E>>I8KL@KFLJCP<M@Ca

$EF;=FIK?@J@J@E;<<;K?<:8J<

`$JK?8KI<8CCP>F@E>KFNFIBK?FL>?aJ8PJLKI<P

$ EF; 8>8@E ?FG@E> K?8K $cD I@>?K $cM< ;FE< K?@J 9<=FI< N@K?KNF 9L:B<KJ 8E; @E GI@E:@GC< K?<I< J?FLC; 9< EF ;@==<I<E:< 9<eKN<<E (8IBcJ ?8E; 8E; 8 9L:B<K M<E @= (8IBcJ ?8E; @J @D9L<;

 N@K? K?< elan vital K?8K ;@JK@E>L@J?<J C@M< D8KK<I =IFD ;<8; D8KK<IK?<KI@:BJ?FLC;NFIB8JN<CC8J@=(8IBN<I<8D8I9C<JK8KL<

(8IB @J CFFB@E> 8K ?@J ?8E; 8 9@K LEE<IM<; `.F_ K?< G<99C< ?8J@EK<EK@FE8C@KP8>8@EEFNa

`4<Ga $ J8P `FEcK 8;; 8EP DFI< G<99C<J KF PFLI ?8E; FIK?IFN8N8PK?<FE<PFL?8M<FIPFLcCC9I<8BK?<I@KL8Ca

(8IB EF;J JFC<DECP /?<E ?< I<JLD<J @EJG<:K@E> K?< G<99C< `$LE;<IJK8E; EFN ?FN PFLI =CF:BJ >I<N JF >I<8Ka (8IB J8PJ `2@K?K?< GFN<I F= K?@J 9L:B<K PFL :FLC; B<<G @E KFJJ@E> G<99C<J 8E; K?<J?<<G NFLC; B<<G I<KLIE@E> =IFD K?< =@<C;J 4FL :FLC; JK8IK N@K?

ALJK 8 =<N J?<<G C<K K?<D C<8M< K?<E =@CC K?< 9L:B<K KF K?< 9I@D9<=FI< K?<P I<KLIE<; E; @= K<E;@E> JF D8EP J?<<G >I<N K<;@FLJ

 PFL :FLC; C<K K?<D 8CC C<8M< K?<E <DGKP 8CDFJK 8CC K?< G<99C<J =IFDK?< 9L:B<K JF K?8K FECP 8 =<N I<KLIE<;_ @E:I<8J@E> K?< =CF:BJ 8>8@E

 N?<E @K :8D< K@D< =FI J?<8I@E>_ ;<8I ?<8M<EJ D8E F PFL I<8C@Q<K?< J?<<I power F= K?@J I@KL8C PFLcM< ;@J:FM<I<; $ :8E FECP @D8>@E<K?< @DGC@:8K@FEJ ?LD8EB@E; D@>?K C<8G 8?<8; 8 ;<:8;< f EF 8 :<EeKLIPa

`$K ;F<JEcK NFIB K?8K N8Pa $ J8P `$= PFL 8;; 8 G<99C< N?<E 8 J?<<G ?8JEcK C<=K FI I<DFM< 8 G<99C< N?<E 8 J?<<G ?8JEcK :FD< @E

%(&,**!,'*1

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 17/309

K?8K 9I<8BJ K?< I@KL8C /?< GFN<I ;F<J EFK C@E><I @E K?< G<99C<J 9LK M8E@J?<J8CC8KFE:<C@B<8JF8G9L99C<GFGG@E>a

(8IBcJ=8:<@JK<II@9CP;@J8GGF@EK<;`I<PFLJLI<a

$EF;`$KI@<;K?8K8E;@K;@;EcKNFIBa(8IB J@>?J ?<8M@CP `E; K?@J_ math_ J<<D<; JF GFN<I=LC 8E;

LJ<=LCLEK@CK?<E_*?N<CC.FDL:?=FI?LD8EGIF>I<JJa

`(8IB @K N8J 8  brilliant @;<8a LKI<P J8PJ <E:FLI8>@E>CP `/?<EFK@FE ;@;EcK F::LI KF D< 8E; P<K @KcJ JF F9M@FLJ_ @K NFLC; J8M< 8Eenormous 8DFLEK F= <==FIK_ K?<I< must 9< 8 N8P KF J8CM8>< PFLI GC8E2< :FLC; KIP ;@==<I<EK 9L:B<KJ CFFB@E> =FI FE< K?8K NFLC; B<<GK?< D8>@:8C GFNe K?< @EK<EK@FE8C@KP @E K?< G<99C<J <M<E N@K?FLK K?<

I@KL8C *I KIP FK?<I G<99C<J (8P9< FLI G<99C<J ALJK ?8M< K?< NIFE> GIFG<IK@<J KF ?8M< inherent @EK<EK@FE8C@KP 2?8K @= N< KI@<; @K LJ@E> JKFE<J :8IM<; KF I<J<D9C< K@EP J?<<G *I ALJK NI@K< bJ?<<Gc FE K?<G<99C<JK?8KD@>?K9<<EFL>?a

`)FK>F@E>KFNFIBa$GI<;@:K;IPCP

 LKI<P :FEK@EL<J `(8P9< N< E<<; FI>8E@: G<99C<J @EJK<8; F= J@C@:FE G<99C<J_ FI D8P9< N< E<<; KF LJ< <OG<EJ@M< ><DJKFE<J/?< GI@:< F= ><DJKFE<J ;FL9C<J <M<IP <@>?K<<E DFEK?J JF PFL:FLC; 9LP 8 ?8E;=LC F= :?<8G ><DJKFE<J EFN 8E; N8@K 8E; @E KN<Ee

KPP<8IJK?<Pc;9<I<8CCP<OG<EJ@M<a`4FL KI@<; 8;;@E> G<99C<J KF :I<8K< DFI< J?<<G 8E; @K ;@;EcK

 NFIBa(8IB8JBJD<`2?8K<O8:KCP;@;PFL;Fa

`$ KFFB 8 ?8E;=LC F= ;FCC8I 9@CCJ /?<E $ ?@; K?< ;FCC8I 9@CCJ LE;<I8 =FC; F= DP 9C8EB<K FE< 9P FE< <8:? K@D< $ ?@; 8EFK?<I 9@CC $KFFB 8EFK?<I G8G<I:C@G =IFD 8 9FO D8B@E> 8 JD8CC ?<8G $ N8J :8I<e=LC EFK KF B<<G KI8:B @E DP ?<8; JF K?8K 8CC $ BE<N N8J K?8K K?<I<

 N<I< bD8EPc ;FCC8I 9@CCJ 8E; bD8EPc G8G<I:C@GJ /?<E N?<E 8CC K?<

9@CCJ N<I< ?@;;<E LE;<I DP 9C8EB<K $ 8;;<; 8 J@E>C< 8;;@K@FE8C G8eG<I:C@G KF K?< ?<8G K?< <HL@M8C<EK F= KFJJ@E> 8E <OKI8 G<99C< @EKFK?< 9L:B<K /?<E $ JK8IK<; K8B@E> ;FCC8I 9@CCJ =IFD LE;<I K?< =FC;8E; GLKK@E> K?< G8G<I:C@GJ 98:B @EKF K?< 9FO 2?<E $ =@E@J?<; 8 J@E>C<G8G<I:C@GN8JC<=KFM<Ia

`2?8K;F<JK?8KI<JLCKD<8Ea8JBJLKI<P

`$K D<8EJ K?< KI@:B ;@;EcK NFIB *E:< $ 9IFB< I@KL8C 9P K?8K J@Ee >C< D@JJK<G K?< GFN<I ;@; EFK C@E><I 9LK M8E@J?<; @EJK8EKCP K?<?<8G F= G8G<I:C@GJ 8E; K?< G@C< F= ;FCC8I 9@CCJ EF CFE><I N<EK <DGKP 

8KK?<J8D<K@D<a

,+!%($,*-,

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 18/309

`4FL actually KI@<;K?@Ja8JBJ(8IB

`4<Ja $ J8P `$ 8:KL8CCP G<I=FID<; K?< <OG<I@D<EK KF M<I@=P K?8KK?< FLK:FD< D8K:?<; DP K?<FI<K@:8C GI<;@:K@FE $ ?8M< 8 J<EK@D<Ee

K8C =FE;E<JJ =FI K?< J:@<EK@=@: D<K?F; <M<E N?<E @K J<<DJ 89JLI;<J@;<JN?8K@=$c;9<<ENIFE>a

`$= @K had  NFIB<;a J8PJ (8IB `PFL NFLC; ?8M< 9<<E >L@CKP F= :FLEK<I=<@K@E> $D8>@E< @= <M<IPFE< ;@; K?8K K?< <:FEFDP NFLC;:FCC8GJ< M<IPFE< NFLC; ?8M< 9@CC@FEJ F= ;FCC8IJ F= :LII<E:P P<KK?<I<NFLC;9<EFK?@E>=FIDFE<PKF9LPa

`)FK 8K 8CCa $ I<GCP `P K?8K J8D< CF>@: N?<I<9P 8;;@E> 8EFK?<IG8G<I:C@G KF K?< ?<8G :I<8K<J 8EFK?<I ;FCC8I 9@CC :I<8K@E> 8EFK?<I;FCC8I 9@CC NFLC; :I<8K< 8E 8;;@K@FE8C ;FCC8IcJ NFIK? F= >FF;J 8E;J<IM@:<Ja

(8IB J?8B<J ?@J ?<8; `FLEK<I=<@K@E> @J JK@CC 8 :I@D<_ 4FLJ?FLC;EFK?8M<KI@<;a

`$N8J reasonably :FE=@;<EK$NFLC;=8@Ca

`?8a J8PJ (8IB `4FL expected  KF =8@C 4FL ;@;EcK believe PFL:FLC;;F@Ka

`$E;<<;a $ 8;D@K `4FL ?8M< >L<JJ<; DP <OG<:K8K@FEJ N@K?JKLEE@E>8::LI8:Pa

`2<CC K?8KcJ K?< GIF9C<Da (8IB J8PJ 9I@JBCP `(8>@: @J =L<C<;9P 9<C@<= 8E; N@CCGFN<I $= PFL ;FEcK 9<C@<M< PFL :8E ;F @K PFL :8EcK4FL E<<; KF :?8E>< PFLI 9<C@<= 89FLK K?< <OG<I@D<EK8C I<JLCK K?8K

 N@CC:?8E><K?<I<JLCK@KJ<C=a

`!LEEPa $ J8P EFJK8C>@:8CCP `K?8KcJ N?8K LKI<P J8@; N?<E $ KFC;?@D 89FLK K?< G<99C<e8E;e9L:B<K D<K?F; /?8K @K N8J KFF I@;@:LeCFLJ=FI?@DKF9<C@<M<JF@KNFLC;EcKNFIB=FI?@Da

`#FN;@;PFLG<IJL8;<?@Da@EHL@I<J(8IB

`$ KFC; ?@D KF J?LK LG 8E; =FCCFN @EJKIL:K@FEJa $ J8P `8E; N?<EK?<D<K?F;NFIB<;LKI<PJK8IK<;9<C@<M@E>@E@Ka

(8IB =IFNEJ GLQQC<; `/?8K D8B<J EF J<EJ< $K ;F<JEcK I<JFCM<K?<<JJ<EK@8C:?@:B<Ee8E;e<>>;@C<DD8a

`.LI< @K ;F<J /?< 9L:B<K D<K?F; NFIBJ N?<K?<I FI EFK PFL 9<eC@<M<@E@Ka

`/?8KcJ absurd! a JGLKK<IJ (8IB `$ ;FEcK 9<C@<M< @E D8>@: K?8K NFIBJN?<K?<IFIEFKPFL9<C@<M<@E@Ka

`$J8@;K?8KKFFa:?@D<J@ELKI<P`GG8I<EKCP$N8JNIFE>a

%(&,**!,'*1

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 19/309

(8IB J:I<NJ LG ?@J =8:< @E :FE:<EKI8K@FE `LK_ @= PFL ;@;EcK9<C@<M< @E D8>@: K?8K NFIBJ N?<K?<I FI EFK PFL 9<C@<M< @E @K K?<E

 N?P ;@; K?< 9L:B<K D<K?F; NFIB N?<E PFL ;@;EcK 9<C@<M< @E @K @;

 PFL 9<C@<M< @E D8>@: K?8K NFIBJ N?<K?<I FI EFK PFL 9<C@<M< @E @K N?<K?<I FI EFK PFL 9<C@<M< @E D8>@: K?8K NFIBJ N?<K?<I FI EFK PFL9<C@<M<@E@Ka

`$;FEcK_ think JF_aJ8PJLKI<P;FL9K=LCCP

`/?<E @= PFL ;@;EcK 9<C@<M< @E D8>@: K?8K NFIBJ N?<K?<I FI EFK PFL_ ?FC; FE 8 J<:FE; $ E<<; KF NFIB K?@J FLK FE G8G<I 8E; G<Ee:@C ea (8IB J:I@99C<J =I8EK@:8CCP CFFBJ JB<GK@:8CCP 8K K?< I<JLCK KLIEJK?< G@<:< F= G8G<I LGJ@;< ;FNE K?<E >@M<J LG `)<M<I D@E;a J8PJ(8IB `(8>@: @J ;@==@:LCK <EFL>? =FI D< KF :FDGI<?<E; D<K8D8>@:

@JFLKF=DP;<GK?a`(8IB $ ;FEcK K?@EB PFL LE;<IJK8E; K?< 8IK F= 9L:B<K:I8=Ka $

J8P `$KcJ EFK 89FLK LJ@E> G<99C<J KF :FEKIFC J?<<G $KcJ 89FLK D8Be@E> J?<<G :FEKIFC G<99C<J $E K?@J 8IK @K @J EFK E<:<JJ8IP KF 9<>@E9P 9<C@<M@E> K?< 8IK N@CC NFIB -8K?<I =@IJK K?< 8IK NFIBJ K?<E FE<:FD<JKF9<C@<M<K?8K@KNFIBJa

`*IJFPFL9<C@<M<aJ8PJ(8IB

`.F $ 9<C@<M<a $ I<GCP `because @K ?8GG<EJ KF 9< 8 =8:K /?<

:FII<JGFE;<E:< 9<KN<<E I<8C@KP 8E; DP 9<C@<=J :FD<J =IFD I<8C@KP :FEKIFCC@E>DP9<C@<=JEFKK?<FK?<IN8P8IFLE;a

 EFK?<IJ?<<GG8JJ<J:8LJ@E>D<KFKFJJ@E8EFK?<IG<99C<

`? )FN N< :FD< KF K?< IFFK F= K?< GIF9C<Da J8PJ (8IB`2?8KcJ K?@J JFe:8CC<; bI<8C@KPc 9LJ@E<JJ $ LE;<IJK8E; N?8K @K D<8EJ=FI 8 ?PGFK?<J@J KF 9< <C<>8EK FI =8CJ@=@89C< FI :FDG8K@9C< N@K? K?<<M@;<E:< $K JFLE;J KF D< C@B< :8CC@E> 8 9<C@<= bKIL<c FI bI<8Cc FI b8:eKL8Cc @J D<I<CP K?< ;@==<I<E:< 9<KN<<E J8P@E> PFL 9<C@<M< JFD<K?@E>8E;J8P@E>PFLI<8CCPI<8CCP9<C@<M<JFD<K?@E>a

$ G8LJ< `2<CC_a $ J8P JCFNCP `!I8EBCP $cD EFK <EK@I<CP JLI<DPJ<C= N?<I< K?@J bI<8C@KPc 9LJ@E<JJ :FD<J =IFD $ :8EcK :I<8K< DP FNE I<8C@KP @E K?< C89 JF $ DLJK EFK LE;<IJK8E; @K P<K LK F::8eJ@FE8CCP $ 9<C@<M< JKIFE>CP K?8K JFD<K?@E> @J >F@E> KF ?8GG<E 8E;K?<E JFD<K?@E> <CJ< ?8GG<EJ @EJK<8; $ E<<; 8 E8D< =FI N?8K<M<Ie@Ke@J K?8K ;<K<ID@E<J DP <OG<I@D<EK8C I<JLCKJ JF $ :8CC @K bI<8C@KPc/?@J bI<8C@KPc @J JFD<?FN J<G8I8K< =IFD <M<E DP M<IP 9<JK ?PGFK?<eJ<J M<E N?<E $ ?8M< 8 J@DGC< ?PGFK?<J@J JKIFE>CP JLGGFIK<; 9P 8CC

K?< <M@;<E:< $ BEFN JFD<K@D<J $cD JK@CC JLIGI@J<; .F $ E<<; ;@==<Ie

,+!%($,*-,

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 20/309

<EK E8D<J =FI K?< K?@E>@<J K?8K ;<K<ID@E< DP GI<;@:K@FEJ 8E; K?<K?@E>P K?8K ;<K<ID@E<J DP <OG<I@D<EK8C I<JLCKJ $ :8CC K?< =FID<IK?@E>@<Jb9<C@<=c8E;K?<C8KK<IK?@E>PbI<8C@KPca

(8IB JEFIKJ `$ ;FEcK <M<E BEFN N?P $ 9FK?<I C@JK<E@E> KF K?@JF9M@FLJ EFEJ<EJ< 2?8K<M<I PFL J8P 89FLK K?@J JFe:8CC<; bI<8C@KPc @K@J D<I<CP 8EFK?<I 9<C@<= M<E PFLI 9<C@<= K?8K I<8C@KP GI<:<;<J PFLI9<C@<=J @J 8 9<C@<= $K =FCCFNJ 8J 8 CF>@:8C @E<M@K89@C@KP K?8K I<8C@KP ;F<JEFK<O@JKFECP9<C@<=J<O@JKa

`#FC; FEa J8PJ LKI<P `:FLC; PFL I<G<8K K?8K C8JK G8IK 4FLCFJKD<N@K?K?8KJ?8IGJN<IM<K?<I<@EK?<D@;;C<a

`)F D8KK<I N?8K PFL J8P 89FLK I<8C@KP @KcJ ALJK 8EFK?<I 9<C@<=a<OGC8@EJ (8IB `$K =FCCFNJ N@K? :ILJ?@E> E<:<JJ@KP K?8K K?<I< @J EFI<8C@KPFECP9<C@<=Ja

`$ J<<a $ J8P `/?< J8D< N8P K?8K EF D8KK<I N?8K PFL <8K PFLE<<; KF <8K @K N@K? PFLI DFLK? $K =FCCFNJ K?8K K?<I< @J EF =FF; FECP DFLK?Ja

`+I<:@J<CPa J8PJ (8IB `M<IPK?@E> K?8K PFL <8K ?8J KF 9< @E PFLI DFLK? #FN :8E K?<I< 9< =FF; K?8K <O@JKJ FLKJ@;< PFLI DFLK?/?< K?FL>?K @J EFEJ<EJ< GIFM@E> K?8K b=FF;c @J 8E @E:F?<I<EK EFeK@FE/?8KcJN?PN<cI<8CCJK8IM@E>KF;<8K?K?<I<cJEF=FF;a

 LKI<P CFFBJ ;FNE 8K ?@J JKFD8:? `LK $cD not  JK8IM@E> KF;<8K?a

 Aha! a J?FLKJ (8IB KI@LDG?8EKCP `E; ?FN ;@; PFL LKK<I K?8K M<IP F9A<:K@FE 2@K? PFLI mouth DP =I@<E; 2@K? PFLI mouth2?8K9<KK<I;<DFEJKI8K@FE:FLC;PFL8JBK?8KK?<I<@JEF=FF;a

`What’s this about starvation? a ;<D8E;J 8 ?8IJ? I8JG@E> MF@:<=IFD ;@I<:KCP 9<?@E; LJ LKI<P 8E; $ JK8P :8CD ?8M@E> >FE<K?IFL>? K?@J 9<=FI< (8IB C<8GJ 8 =FFK @E K?< 8@I JK8IKC<; 8CDFJK FLK

F=?@JN@KJ$EJG<:KFI 8IN@E JD@C<J K@>?KCP GC<8J<; 8K 8:?@<M@E> JLIGI@J<8E;D8B<J8JD8CCK@:BFE?@J:C@G9F8I;

`%LJK 8 D<K8G?FIa (8IB J8PJ HL@:BCP `4FL ;FEcK E<<; KF K8B<8N8PDPDFLK?FI8EPK?@E>C@B<K?8Kea

`Why ;F PFL E<<; 8 mouth @= K?<I< @J EF food a ;<D8E;J 8IN@E8E>I@CP  Never mind. $ ?8M< EF time =FI K?@J foolishness $ 8D ?<I< KF@EJG<:KK?< sheep.a

`!CF:BJK?I@M@E>J@Ia$J8P`)F;<8;J?<<GJ@E:<%8EL8IPa

%(&,**!,'*1

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 21/309

 Excellent. $ 8N8I; PFL LE@KJ F= fitness )FN N?8K @J K?@J per- son ;F@E>?<I<$J?<8E<:<JJ8IPG8IKF=K?< operations? a

`J =8I 8J $ :8E J<< ?< NFLC; 9< F= DFI< LJ< KF K?< ?LD8E

JG<:@<J@=?LE>F==8?FKe8@I98CCFFE8J98CC8JKa$J8P`*L:?aJ8PJLKI<PD@C;CP

`$ ;F EFK care 89FLK K?< human species '<K ?@D JG<8B =FI himself a

(8IB ;I8NJ ?@DJ<C= LG ?8L>?K@CP `/?@J D<I< shepherd a ?< J8PJ ><JKLI@E> 8K D< `?8J :C8@D<; K?8K K?<I< @J JL:? 8 K?@E> 8J I<8C@KP/?@J F==<E;J D< =FI $ BEFN N@K? ;<<G 8E; 89@;@E> :<IK8@EKP K?8KK?<I< @J EF KILK? /?< :FE:<GK F= bKILK?c @J D<I<CP 8 JKI8K8><D =FIG<FGC< KF @DGFJ< K?<@I FNE 9<C@<=J FE FK?<IJ M<IP :LCKLI< ?8J 8 

;@==<I<EK bKILK?c 8E; EF :LCKLI<cJ bKILK?c @J JLG<I@FI KF 8EP FK?<I/?@J K?8K $ ?8M< J8@; ?FC;J 8K 8CC K@D<J @E 8CC GC8:<J 8E; $ @EJ@JK K?8K PFL8>I<<a

`#FC; FE 8 J<:FE;a J8PJ LKI<P `$= EFK?@E> @J KIL< N?P J?FLC;$9<C@<M<PFLN?<EPFLJ8PK?8KEFK?@E>@JKIL<a

`$;@;EcKJ8PK?8KEFK?@E>@JKIL<eaJ8PJ(8IB

`4<JPFL;@;a@EK<IA<:KJLKI<P`$?<8I;PFLa

`e $ J8@; K?8K bKILK?c @J 8E <O:LJ< LJ<; 9P JFD< :LCKLI<J KF <Ee=FI:< K?<@I 9<C@<=J FE FK?<IJ .F N?<E PFL J8P JFD<K?@E> @J bKIL<c

 PFL D<8E FECP K?8K @K NFLC; 9< 8;M8EK8><FLJ KF PFLI FNE JF:@8C >IFLGKF?8M<@K9<C@<M<;a

`E;K?@JK?8KPFL?8M<J8@;a$J8P`@J@KKIL<a

`9JFCLK<CP GFJ@K@M<CP KIL<a J8PJ (8IB <DG?8K@:8CCP `+<FGC<:I<8K<K?<@IFNEI<8C@K@<Ja

`#FC; FEa J8PJ LKI<P JFLE;@E> GLQQC<; 8>8@E `J8P@E> K?8KG<FGC< :I<8K< K?<@I FNE I<8C@K@<J @J CF>@:8CCP 8 :FDGC<K<CP J<G8I8K<@JJL< =IFD J8P@E> K?8K K?<I< @J EF KILK? 8 JK8K< F= 8==8@IJ $ :8EEFK

<M<E @D8>@E< :F?<I<EKCP G<I?8GJ 9<:8LJ< PFL JK@CC ?8M< EFK <OeGC8@E<;?FN<O8:KCP@K@JJLGGFJ<;KFNFIBea

`/?<I< PFL >F 8>8@Ea J8PJ (8IB <O8JG<I8K<;CP `KIP@E> KF 8GGCP  PFLI 2<JK<IE :FE:<GKJ F= CF>@: I8K@FE8C@KP I<8JFE :F?<I<E:< 8E;J<C=e:FEJ@JK<E:Pa

`"I<8Ka DLKK<IJ LKI<P `EFN $ E<<; KF 8;; 8  third  JL9A<:K?<8;@E> KF B<<G KI8:B F= K?@J <EK@I<CP J<G8I8K< 8E; ;@JK@E:K :C8@D ea

`$KcJ EFK J<G8I8K<a J8PJ (8IB `'FFB PFLcI< K8B@E> K?< NIFE> 

8KK@KL;< 9P KI<8K@E> DP JK8K<D<EKJ 8J ?PGFK?<J<J 8E; :8I<=LCCP 

,+!%($,*-,

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 22/309

;<I@M@E> K?<@I :FEJ<HL<E:<J 4FL E<<; KF K?@EB F= K?<D 8J =LCCP  ><E<I8C <O:LJ<J N?@:? $ 8GGCP N?<E 8EPFE< J8PJ JFD<K?@E> $ ;FEcKC@B< $KcJ EFK JF DL:? 8 DF;<C F= ?FN K?< LE@M<IJ< NFIBJ 8J 8 `"<K

*LK F= %8@C !I<<a :8I; /?< key @J KF 8GGCP K?< <O:LJ< selectively2?<E $ J8P K?8K K?<I< @J EF JL:? K?@E> 8J KILK? K?8K 8GGC@<J FECP KF your :C8@D K?8K K?< D8>@: 9L:B<K NFIBJ N?<K?<I FI EFK $ 9<C@<M< @E@K $K ;F<J not 8GGCP KF my :C8@D K?8K K?<I< @J EF JL:? K?@E> 8J KILK?a

`0D_N?PEFKa@EHL@I<JLKI<P

(8IB ?<8M<J 8 G8K@<EK J@>? `LKI<P ;F PFL K?@EB PFLcI< K?<=@IJK G<IJFE KF K?@EB F= K?8K HL<JK@FE /F 8JB LJ ?FN FLI FNE 9<eC@<=J :8E 9< D<8E@E>=LC @= 8CC 9<C@<=J 8I< D<8E@E>C<JJ /?8KcJ K?<J8D< K?@E> D8EP JKL;<EKJ J8P N?<E K?<P <E:FLEK<I K?@J G?@CFJFG?P

 N?@:? $cCC ?8M< PFL BEFN ?8J D8EP 8;?<I<EKJ 8E; 8E <OK<EJ@M< C@Ke<I8KLI<a

`.FN?8KcJK?<8EJN<IaJ8PJLKI<P

`2<E8D<;@KK?<bI<=C<O@M@KPGIF9C<Dca<OGC8@EJ(8IB

`LKN?8KcJK?< answer aG<IJ@JKJLKI<P

(8IB JD@C<J :FE;<J:<E;@E>CP `<C@<M< D< LKI<P PFLcI< EFKK?< =@IJK G<IJFE KF K?@EB F= JL:? 8 J@DGC< HL<JK@FE /?<I<cJ EF GF@EK@EGI<J<EK@E>@KKFLJ8J8KI@LDG?8EKI<=LK8K@FEa

`LKN?8KcJK?< actual answer? a`)FN $c; C@B< KF DFM< FE KF K?< @JJL< F= ?FN CF>@: B@CCJ :LK< 98e

9PJ<8CJea

`You 8I<N8JK@E> timeaJE8GJ$EJG<:KFI8IN@E

`)FK KF D<EK@FE CFJ@E> KI8:B F= J?<<Ga $ J8P KFJJ@E> @E 8EFK?<IG<99C<

$EJG<:KFI 8IN@E CFFBJ 8K K?< KNF 8I>L<IJ 9FK? 8GG8I<EKCP LEe N@CC@E> KF >@M< LG K?<@I GFJ@K@FEJ `'@JK<Ea 8IN@E J8PJ DFI< B@E;CP 

EFN `$ ?8M< 8 J@DGC< EFK@FE =FI I<JFCM@E> PFLI ;@JGLK< You J8PaJ8PJ 8IN@E GF@EK@E> KF (8IB `K?8K G<FGC<cJ 9<C@<=J 8CK<I K?<@I G<IeJFE8C I<8C@K@<J E; you =<IM<EKCP 9<C@<M<a ?@J =@E><I JN@M<CJ KF GF@EK8K LKI<P `K?8K (8IBcJ 9<C@<=J can’t 8CK<I I<8C@KP .F C<K (8IB 9<C@<M<I<8CCP ?8I; K?8K ?< :8E =CP 8E; K?<E JK<G F== 8 :C@== (8IB J?8CC J<<?@DJ<C= =CP 8N8P C@B< 8 9@I; 8E; LKI<P J?8CC J<< ?@D GCLDD<K ;FNE8E;>FJGC8K8E;PFLJ?8CC9FK?9<?8GGPa

2<8CCG8LJ<:FEJ@;<I@E>K?@J

`$K sounds I<8JFE89C<_a(8IBJ8PJ=@E8CCP

%(&,**!,'*1

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 23/309

`/?<I<cJ8:C@==I@>?KK?<I<aF9J<IM<J$EJG<:KFI8IN@E

 LKI<P @J N<8I@E> 8 CFFB F= @EK<EJ< :FE:<EKI8K@FE !@E8CCP ?<J?FLKJ `28@K $= K?8K N<I< KIL< N< NFLC; 8CC ?8M< CFE> J@E:< ;<e

G8IK<; @EKF FLI FNE GI@M8K< LE@M<IJ<J @E N?@:? :8J< K?< FK?<IG<FGC< ?<I< 8I< FECP =@>D<EKJ F= PFLI @D8>@E8K@FE f K?<I<cJ EF GF@EK@EKIP@E>KFGIFM<8EPK?@E>KFLJea

  CFE> ;N@E;C@E> J:I<8D :FD<J =IFD K?< E<8I9P :C@== =FCCFN<;9P 8 ;LCC 8E; CFE<CP JGC8K $EJG<:KFI 8IN@E =C@GJ ?@J :C@G9F8I; KFK?< G8>< K?8K J?FNJ K?< :LII<EK ><E< GFFC 8E; G<E:@CJ @E 8 JC@>?KCP CFN<I=I<HL<E:P=FI(8IBcJ8CC<C<J

 LKI<PCFFBJJC@>?KCPJ@:B`28JK?8KI<8CCPE<:<JJ8IPa

 Necessary? a J8PJ $EJG<:KFI 8IN@E JFLE;@E> GLQQC<; `$K ALJK happened _$;FEcKHL@K<LE;<IJK8E;PFLIHL<JK@FEa

 LKI<P 8E; $ KLIE 98:B KF FLI 9L:B<K $KcJ K@D< KF 9I@E> @EK?< J?<<G 4FL NFLC;EcK N8EK KF =FI><K 89FLK K?8K G8IK *K?<IN@J<

 N?8KNFLC;9<K?<GF@EK

,+!%($,*-,

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 24/309

2. What Do We Mean By “Rationality”?

2<D<8E

Epistemic rationality 9<C@<M@E>8E;LG;8K@E>FE<M@;<E:<JF8JKFJPJK<D8K@:8CCP@DGIFM<K?<:FII<JGFE;<E:<9<KN<<E PFLID8G8E;K?<K<II@KFIP /?<8IKF=F9K8@E@E>9<C@<=JK?8K:FII<JGFE;KFI<8C@KP8J:CFJ<CP8JGFJJ@9C< /?@J:FII<JGFE;<E:<@J:FDDFECP K<ID<;`KILK?aFI`8::LI8:Pa8E;N<cI<?8GGPKF:8CC@KK?8K

Instrumental rationality 8:?@<M@E>PFLIM8CL<J Not 

E<:<JJ8I@CP`PFLIM8CL<Ja@EK?<J<EJ<F=9<@E> selfish M8CL<JFI unshared M8CL<J`PFLIM8CL<JaD<8EJ anything you care about  /?<8IKF=:?FFJ@E>8:K@FEJK?8KJK<<IK?<=LKLI<KFN8I;FLK:FD<JI8EB<;?@>?<I@EPFLIGI<=<I<E:<J *E'2N<JFD<K@D<JI<=<IKFK?@J8J`N@EE@E>a

$= K?8K J<<DJ C@B< 8 G<I=<:KCP >FF; ;<=@E@K@FE PFL :8E JKFG I<8;e@E>?<I<FK?<IN@J<:FEK@EL<

.FD<K@D<J <OG<I@D<EK8C GJP:?FCF>@JKJ LE:FM<I ?LD8E I<8JFEe@E> K?8K J<<DJ M<IP JKI8E>< e =FI <O8DGC< JFD<FE< I8K<J K?< GIF9e

89@C@KP `@CC GC8PJ A8QQa 8J less K?8E K?< GIF989@C@KP `@CC @J 8E 8::FLEeK8EK N?F GC8PJ A8QQa /?@J J<<DJ C@B< 8E F;; AL;>D<EK J@E:< 8EP G8IK@:LC8I A8QQeGC8P@E> 8::FLEK8EK @J F9M@FLJCP 8 A8QQ GC8P<I LK KF

 N?8K ?@>?<I M8EK8>< GF@EK ;F N< 8GG<8C @E J8P@E> K?8K K?< AL;>D<EK@J wrong? 

OG<I@D<EK8C GJP:?FCF>@JKJ LJ< KNF >FC; JK8E;8I;J probabilitytheory, 8E; decision theory .@E:< @K @J 8 LE@M<IJ8C C8N F= GIF989@C@KP K?<FIP K?8K +hi ^ +h i K?< AL;>D<EK +h`@CC GC8PJ A8QQai +h`@CCGC8PJA8QQa`@CC@J8::FLEK8EKai@JC89<C<;@E:FII<:K

/F B<<G @K K<:?E@:8C PFL NFLC; J8P K?8K K?@J GIF989@C@KP AL;>eD<EK @J non-Bayesian. <C@<=J K?8K :FE=FID KF 8 :F?<I<EK GIF989@C@KP ;@JKI@9LK@FE 8E; ;<:@J@FEJ K?8K D8O@D@Q< K?< GIF989@C@JK@: <OG<:K8eK@FEF=8:F?<I<EKLK@C@KP=LE:K@FE8I<:8CC<;`8P<J@8Ea

/?@J ;F<J EFK HL@K< <O?8LJK K?< GIF9C<D F= N?8K @J D<8EK @EGI8:K@:<9P`I8K@FE8C@KPa=FIKNFD8AFII<8JFEJ

http://lesswrong.com/lw/31/what_do_we_mean_by_rationality/ 

+8><b/?<.@DGC</ILK?c http://lesswrong.com/lw/ji/conjunction_fallacy/ 

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 25/309

!@IJK K?< 8P<J@8E =FID8C@JDJ @E K?<@I =LCC =FID 8I< :FDGLeK8K@FE8CCP @EKI8:K89C< FE DFJK I<8CeNFIC; GIF9C<DJ )F FE< :8E actually :8C:LC8K< 8E; F9<P K?< D8K? 8EP DFI< K?8E PFL :8E GI<;@:K

K?<JKF:BD8IB<K9P:8C:LC8K@E>K?<DFM<D<EKJF=HL8IBJ/?@J @J N?P N< ?8M< 8 N?FC< J@K< :8CC<; `'<JJ 2IFE>a I8K?<I

K?8E J@DGCP JK8K@E> K?< =FID8C 8O@FDJ 8E; 9<@E> ;FE< /?<I<cJ8 N?FC< =LIK?<I 8IK KF =@E;@E> K?< KILK? 8E; 8::FDGC@J?@E> M8CL< from inside a human mind: N< ?8M< KF C<8IE FLI FNE =C8NJ FM<I:FD<FLI 9@8J<J GI<M<EK FLIJ<CM<J =IFD J<C=e;<:<@M@E> ><K FLIJ<CM<J @EKF

 >FF; <DFK@FE8C J?8G< KF :FE=IFEK K?< KILK? 8E; ;F N?8K E<<;J ;Fe@E><K:<K<I8<K:<K<I88E;JFFE

.<:FE; JFD<K@D<J K?< D<8E@E> F= K?< D8K? @KJ<C= @J :8CC<; @EKF

HL<JK@FE /?< <O8:K ILC<J F= GIF989@C@KP K?<FIP 8I< :8CC<; @EKF HL<JeK@FE 9P <> 8EK?IFG@: GIF9C<DJ @E N?@:? K?< ELD9<I F= F9J<IM<IJ@J LE:<IK8@E /?< <O8:K ILC<J F= ;<:@J@FE K?<FIP 8I< :8CC<; @EKF HL<JeK@FE 9P <> )<N:FD9C@B< GIF9C<DJ @E N?@:? FK?<I 8><EKJ D8P GI<;@:KPFLI;<:@J@FE9<=FI<@K?8GG<EJ

$E :8J<J C@B< K?<J< @K @J =LK@C< KF KIP KF J<KKC< K?< GIF9C<D 9P :FD@E> LG N@K? JFD< E<N ;<=@E@K@FE F= K?< NFI; `I8K@FE8Ca 8E; J8Pe@E> `/?<I<=FI< DP GI<=<II<; 8EJN<I by definition, @J N?8K @J D<8EK

9P K?< NFI; bI8K@FE8Cca /?@J J@DGCP 9<>J K?< HL<JK@FE F= N?P 8EPeFE< J?FLC; G8P 8KK<EK@FE KF PFLI ;<=@E@K@FE 2< 8I<EcK @EK<I<JK<;@E GIF989@C@KP K?<FIP 9<:8LJ< @K @J K?< ?FCP NFI; ?8E;<; ;FNE =IFD'8GC8:< 2<cI< @EK<I<JK<; @E 8P<J@8EeJKPC< 9<C@<=eLG;8K@E> hN@K?*::8D GI@FIJi 9<:8LJ< N< <OG<:K K?8K K?@J JKPC< F= K?@EB@E> ><KJ LJJPJK<D8K@:8CCP :CFJ<I KF PFL BEFN accuracy, K?< D8G K?8K I<=C<:KJK?< K<II@KFIP h(FI< FE K?< =LK@C@KP F= 8I>L@E> `9P ;<=@E@K@FEa ?<I<

8E; ?<I<i

 E; K?<E K?<I< 8I< HL<JK@FEJ F= `#FN KF K?@EBa K?8K J<<D EFKHL@K< 8EJN<I<; 9P <@K?<I GIF989@C@KP K?<FIP FI ;<:@J@FE K?<FIP e C@B<K?< HL<JK@FE F= ?FN KF =<<C 89FLK K?< KILK? FE:< N< ?8M< @K #<I<8>8@E KIP@E> KF ;<=@E< `I8K@FE8C@KPa 8 G8IK@:LC8I N8P ;F<JEcK JLGGFIK8E8EJN<ID<I<CPGI<JLD<@K

!IFDK?< /N<CM<1@IKL<JF=-8K@FE8C@KP  :

http://www.anthropic-principle.com/primer.html  http://lesswrong.com/lw/nc/newcombs_problem_and_regret_of_rationality/  +8><b/?<+8I89C<F=#<DCF:Bc

+8><bI>L@E>`P<=@E@K@FEac +8><b!<<C@E>-8K@FE8Cc

/,'/%&14* ,!'&$!,15

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 26/309

#FN:8EPFL@DGIFM<PFLI:FE:<GK@FEF= I8K@FE8C@KP )FK9PJ8P@E>KFPFLIJ<C=`$K@JDP;LKP KF9<I8K@FE8Ca PK?@JPFLFECP<EJ?I@E<PFLI

D@JK8B<E:FE:<GK@FE +<I?8GJPFLI:FE:<GK@FEF= I8K@FE8C@KP@JK?8K@K@JI8K@FE8CKF9<C@<M<K?<NFI;JF=K?<"I<8K/<8:?<I8E;K?<"I<8K/<8:?<IJ8PJ`/?<JBP@J>I<<Ea8E;PFLCFFBLG8KK?<JBP8E;J<<9CL< $=PFLK?@EB `$KD8PCFFBC@B<K?<JBP@J9CL<9LKI8K@FE8C@KP@JKF9<C@<M<K?<NFI;JF=K?<"I<8K/<8:?<IaPFLCFJ<8:?8E:<KF;@J:FM<IPFLID@JK8B<

FEFK8JBN?<K?<I@K@J`K?<28PaKF;FK?@JFIK?8K JBN?<K?<IK?<JBP@J9CL<FI>I<<E $=PFLJG<8BFM<IDL:?F=K?<28PPFLN@CCEFK8KK8@E@K

4FLD8PKIPKFE8D<K?<?@>?<JKGI@E:@GC<N@K?E8D<JJL:?8J`K?<D8GK?8KI<=C<:KJK?<K<II@KFIPaFI`<OG<I@<E:<F=JL::<JJ8E;=8@CLI<aFI`8P<J@8E;<:@J@FEK?<FIPa LKG<I?8GJPFL;<J:I@9<@E:FII<:KCPK?<E8D<C<JJM@IKL< #FNN@CCPFL

;@J:FM<IPFLID@JK8B< )FK9P:FDG8I@E>PFLI;<J:I@GK@FEKF@KJ<C=9LK9P:FDG8I@E>@KKFK?8K

 N?@:?PFL;@;EFKE8D<

2< 8I< EFK ?<I< KF 8I>L< K?< D<8E@E> F= 8 NFI; EFK <M<E @= K?8K NFI; @J `I8K@FE8C@KPa /?< GF@EK F= 8KK8:?@E> J<HL<E:<J F= C<KeK<IJ KF G8IK@:LC8I :FE:<GKJ @J KF C<K KNF G<FGC< communicate e KF?<CG KI8EJGFIK K?FL>?KJ =IFD FE< D@E; KF 8EFK?<I 4FL :8EEFK:?8E>< I<8C@KP FI GIFM< K?< K?FL>?K 9P D8E@GLC8K@E> N?@:? D<8Ee@E>J>FN@K?N?@:?NFI;J

.F @= PFL LE;<IJK8E; N?8K :FE:<GK N< 8I< generally getting at N@K?K?@J NFI; `I8K@FE8C@KPa 8E; N@K? K?< JL9eK<IDJ `<G@JK<D@: I8K@FE8Ce@KPa 8E; `@EJKILD<EK8C I8K@FE8C@KPa N< have communicated: N< ?8M<8::FDGC@J?<; <M<IPK?@E> K?<I< @J KF 8::FDGC@J? 9P K8CB@E> 89FLK?FN KF ;<=@E< `I8K@FE8C@KPa 2?8KcJ C<=K KF ;@J:LJJ @J EFK what mean-

http://yudkowsky.net/virtues/ 

+8><b@JGLK@E><=@E@K@FEJc +8><b/?<I>LD<EK=IFDFDDFE0J8><c

%(&,**!,'*1

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 27/309

ing KF 8KK8:? KF K?< JPCC89C<J `I8eK@FeE8eC@eKPa N?8KcJ C<=K KF ;@J:LJJ @Jwhat is a good way to think.

2@K? K?8K J8@; PFL J?FLC; 9< 8N8I< K?8K D8EP F= LJ N@CC I<>8I;

8J controversial e 8K K?< M<IP C<8JK e 8EP :FEJKIL8C F= `I8K@FE8C@KPa K?8KD8B<J@K non-normative:

!FI <O8DGC< @= PFL J8P `/?< I8K@FE8C 9<C@<= @J 3 9LK K?< KIL<9<C@<= @J 4a K?<E PFL 8I< GIF989CP LJ@E> K?< NFI; `I8K@FE8Ca @E 8 N8P K?8K D<8EJ JFD<K?@E> FK?<I K?8E N?8K DFJK F= LJ ?8M< @E D@E;h> JFD< F= LJ <OG<:K `I8K@FE8C@KPa KF 9< consistent under reflectione `I8K@FE8CCPa CFFB@E> 8K K?< <M@;<E:< 8E; `I8K@FE8CCPa :FEJ@;<I@E> ?FN PFLI D@E; GIF:<JJ<J K?< <M@;<E:< J?FLC;EcK C<8; KF KNF ;@==<Ie<EK :FE:CLJ@FEJi .@D@C8ICP @= PFL =@E; PFLIJ<C= J8P@E> `/?< I8K@FE8C

K?@E> KF ;F @J 3 9LK K?< I@>?K K?@E> KF ;F @J 4a K?<E PFL 8I< 8CDFJK:<IK8@ECP LJ@E> FE< F= K?< NFI;J `I8K@FE8Ca FI `I@>?Ka @E 8 N8P K?8K 8 ?L><:?LEBF=I<8;<IJNFEcK8>I<<N@K?

$E K?@J :8J< e FI @E 8EP FK?<I :8J< N?<I< :FEKIFM<IJP K?I<8K<EJe PFL J?FLC; JL9JK@KLK< DFI< JG<:@=@: C8E>L8>< `/?< J<C=e9<E<=@Ke@E> K?@E> KF ;F @J KF ILE 8N8P 9LK $ ?FG< $ NFLC; 8K C<8JK KIP KF ;I8> K?< >@IC F== K?< I8@CIF8; KI8:BJa FI `8LJ8C ;<:@J@FE K?<FIP 8J LJL8CCP =FIDLC8K<; J8PJ PFL J?FLC; KNFe9FO FE )<N:FD9cJ +IF9C<D 9LK

$c;I8K?<I?8M<8D@CC@FE;FCC8IJa`3 @J I8K@FE8Ca @J LJL8CCP ALJK 8 DFI< JKI@;<EK N8P F= J8P@E> `$

K?@EB 3 @J KIL<a FI `$ K?@EB 3 @J >FF;a .F N?P ?8M< 8E 8;;@K@FE8C NFI; =FI `I8K@FE8Ca 8J N<CC 8J `KIL<a 8E; `>FF;a <:8LJ< N< N8EKKFK8CB89FLK systematic methods =FIF9K8@E@E>KILK?8E;N@EE@E>

/?< NFI; `I8K@FE8Ca ?8J GFK<EK@8C G@K=8CCJ 9LK K?<I< 8I< GC<EKP F=  none9FI;<IC@E< :8J<J N?<I< `I8K@FE8Ca NFIBJ =@E< KF communicate

 N?8K FE< @J ><KK@E> 8K C@B<N@J< `@II8K@FE8Ca $E K?<J< :8J<J N<cI< EFK8=I8@;KFLJ<@K

4<K FE< J?FLC; 8CJF 9< :8I<=LC EFK KF overuse K?8K NFI; *E<I<:<@M<J EF GF@EKJ D<I<CP =FI GIFEFLE:@E> @K CFL;CP $= PFL JG<8BFM<IDL:?F=K?<28PPFLN@CCEFK8KK8@E@K

+8><b/89FF4FLI2FI;Jc http://lesswrong.com/lw/nc/newcombs_problem_and_regret_of_rationality/ 

/,'/%&14* ,!'&$!,15

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 28/309

3. An Intuitive Explanation of Bayes'Theorem 

 Bayes’ Theorem for the curious and bewildered; an excruciatingly gentle introduction.

4FLI=I@<E;J8E;:FCC<8>L<J8I<K8CB@E>89FLKJFD<K?@E>:8CC<;`8P<Jc/?<FI<DaFI 8P<Jc-LC<aFIJFD<K?@E>:8CC<;8P<J@8EI<8JFE@E> /?<PJFLE;I<8CCP<EK?LJ@8JK@:89FLK@KKFFJFPFL

 >FF>C<8E;=@E;8N<9G8><89FLK8P<Jc/?<FI<D8E;_

$KcJK?@J<HL8K@FE /?8KcJ8CC %LJKFE<<HL8K@FE /?<G8><PFL=FLE;>@M<J8;<=@E@K@FEF=@K9LK@K;F<JEcKJ8PN?8K@K@JFIN?P @KcJLJ<=LCFIN?PPFLI=I@<E;JNFLC;9<@EK<I<JK<;@E@K $KCFFBJC@B<K?@JI8E;FDJK8K@JK@:JK?@E>

.FPFL:8D<?<I< (8P9<PFL;FEcKLE;<IJK8E;N?8KK?<<HL8K@FEJ8PJ (8P9<PFLLE;<IJK8E;@K@EK?<FIP9LK<M<IPK@D<PFLKIPKF8GGCP@K@EGI8:K@:<PFL><KD@O<;LGKIP@E>KFI<D<D9<IK?<;@==<I<E:<9<KN<<E D6PL 8E; DLP68E;N?<K?<ID6DLP6 9<CFE>J@EK?<ELD<I8KFIFIK?<;<EFD@E8KFI(8P9<PFLJ<<K?<K?<FI<D8E;PFLLE;<IJK8E;K?<K?<FI<D8E;

 PFL:8ELJ<K?<K?<FI<D9LKPFL:8EcKLE;<IJK8E;N?PPFLI=I@<E;J8E;FII<J<8I:?:FCC<8>L<JJ<<DKFK?@EB@KcJK?<J<:I<KF= K?<LE@M<IJ< (8P9<PFLI=I@<E;J8I<8CCN<8I@E>8P<Jc/?<FI<D/eJ?@IKJ8E;PFLcI<=<<C@E>C<=KFLK (8P9<PFLcI<8>@ICCFFB@E> =FI89FP=I@<E;9LKK?<9FPPFLcI<@EK<I<JK<;@EI<=LJ<JKF;8K<8EPFE<N?F`@JEcK8P<J@8Ea 2?8KD8KK<IJ@JK?8K8P<J@J:FFC

8E;@=PFL;FEcKBEFN8P<JPFL8I<EcK:FFC

2?P;F<J8D8K?<D8K@:8C:FE:<GK><E<I8K<K?@JJKI8E><<EK?LJ@8JD@E@KJJKL;<EKJ 2?8K@JK?<JFe:8CC<;8P<J@8E-<MFCLK@FEEFNJN<<G@E>K?IFL>?K?<J:@<E:<JN?@:?:C8@DJKFJL9JLD<<M<EK?<<OG<I@D<EK8CD<K?F;@KJ<C=8J8JG<:@8C:8J<2?8K@JK?<J<:I<KK?8KK?<8;?<I<EKJF=8P<JBEFN 2?8K@JK?<C@>?KK?8KK?<P?8M<J<<E

http://yudkowsky.net/rational/bayes

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 29/309

.FFEPFLN@CCBEFN .FFEPFLN@CC9<FE<F=LJ

2?@C<K?<I<8I<8=<N<O@JK@E>FEC@E<<OGC8E8K@FEJF=8P<Jc

/?<FI<DDP<OG<I@<E:<N@K?KIP@E>KF@EKIF;L:<G<FGC<KF8P<J@8EI<8JFE@E>@JK?8KK?<<O@JK@E>FEC@E<<OGC8E8K@FEJ8I<KFF89JKI8:K 8P<J@8EI<8JFE@E>@JM<IP counterintuitive. +<FGC<;FEFK<DGCFP8P<J@8EI<8JFE@E>@EKL@K@M<CP=@E;@KM<IP;@==@:LCKKFC<8IE8P<J@8EI<8JFE@E>N?<EKLKFI<;8E;I8G@;CP=FI><K8P<J@8ED<K?F;JFE:<K?<KLKFI@E>@JFM<I /?@J?FC;J<HL8CCPKIL<=FIEFM@:<JKL;<EKJ8E;?@>?CPKI8@E<;GIF=<JJ@FE8CJ@E8=@<C;8P<J@8EI<8JFE@E>@J8GG8I<EKCPFE<F=K?FJ<K?@E>JN?@:?C@B<HL8EKLDD<:?8E@:JFIK?<28JFE.<C<:K@FE/<JK@J@E?<I<EKCP 

;@==@:LCK=FI?LD8EJKF>I8JGN@K?FLI9L@CKe@ED<EK8C=8:LCK@<J

*IJFK?<P:C8@D #<I<PFLN@CC=@E;8E8KK<DGKKFF==<I8Eintuitive <OGC8E8K@FEF=8P<J@8EI<8JFE@E>e8E<O:IL:@8K@E>CP 

 ><EKC<@EKIF;L:K@FEK?8K@EMFB<J8CCK?<?LD8EN8PJF=>I8JG@E> ELD9<IJ=IFDE8KLI8C=I<HL<E:@<JKFJG8K@8CM@JL8C@Q8K@FE /?<@EK<EK@JKF:FEM<PEFK89JKI8:KILC<J=FID8E@GLC8K@E>ELD9<IJ9LKN?8KK?<ELD9<IJD<8E8E;N?PK?<ILC<J8I<N?8KK?<P8I<h8E;:8EEFKGFJJ@9CP9<8EPK?@E><CJ<i 2?<EPFL8I<=@E@J?<;

I<8;@E>K?@JG8><PFLN@CCJ<<8P<J@8EGIF9C<DJ@EPFLI;I<8DJ

 E;C<KcJ9<>@E

#<I<cJ8JKFIPGIF9C<D89FLK8J@KL8K@FEK?8K;F:KFIJF=K<E<E:FLEK<I

nF=NFD<E8K8><=FIKPN?FG8IK@:@G8K<@EIFLK@E<J:I<<E@E>?8M<9I<8JK:8E:<I nF=NFD<EN@K?9I<8JK:8E:<IN@CC><KGFJ@K@M<D8DDF>I8G?@<J nF=NFD<E

 N@K?FLK9I<8JK:8E:<IN@CC8CJF><KGFJ@K@M<D8DDF>I8G?@<J NFD8E@EK?@J8><>IFLG?8;8GFJ@K@M<D8DDF>I8G?P@E8IFLK@E<J:I<<E@E> 2?8K@JK?<GIF989@C@KPK?8KJ?<8:KL8CCP?8J9I<8JK:8E:<I

2?8K;FPFLK?@EBK?<8EJN<I@J $=PFL?8M<EcK<E:FLEK<I<;K?@J

B@E;F=GIF9C<D9<=FI<GC<8J<K8B<8DFD<EKKF:FD<LGN@K?

 &! &,-!,!.0($&,!'&'1+,'*%

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 30/309

 PFLIFNE8EJN<I9<=FI<:FEK@EL@E>

)<OKJLGGFJ<$KFC;PFLK?8KDFJK;F:KFIJ><KK?<J8D<NIFE> 8EJN<IFEK?@JGIF9C<DeLJL8CCPFECP8IFLE;nF=;F:KFIJ><K@KI@>?K h`-<8CCP n $JK?8K8I<8CELD9<IFI8ELI98EC<><E;98J<;FE8E$EK<IE<KGFCCa $KcJ8I<8CELD9<I .<<8JJ:<CCJ.:?F<E9<I><I8E;"I8P9FPJ;;P"@><I<EQ<I8E;#F==I8><8E;D8EPFK?<IJKL;@<J $KcJ8JLIGI@J@E>I<JLCK

 N?@:?@J<8JPKFI<GC@:8K<JF@KcJ9<<E<OK<EJ@M<CPI<GC@:8K<;i

FPFLN8EKKFK?@EB89FLKPFLI8EJN<I8>8@E #<I<cJ8%8M8J:I@GK

:8C:LC8KFI@=PFLE<<;FE< /?@J:8C:LC8KFI?8JK?<LJL8CGI<:<;<E:<ILC<JDLCK@GC@:8K@FE9<=FI<8;;@K@FE8E;JFFE $= 

 PFLcI<EFKJLI<$JL>><JKLJ@E>G8I<EK?<J<J

*EK?<JKFIPGIF9C<D89FM<DFJK;F:KFIJ<JK@D8K<K?<GIF989@C@KPKF9<9<KN<<En8E;nN?@:?@JN@C;CP@E:FII<:K

#<I<cJ8E8CK<IE8K<M<IJ@FEF=K?<GIF9C<DFEN?@:?;F:KFIJ=8I<JFD<N?8K9<KK<I

FLKF=NFD<E8K8><=FIKPN?FG8IK@:@G8K<@EIFLK@E<J:I<<E@E>?8M<9I<8JK:8E:<I FLKF=

 NFD<EN@K?9I<8JK:8E:<IN@CC><KGFJ@K@M<D8DDF>I8G?@<J FLKF=NFD<EN@K?FLK9I<8JK:8E:<IN@CC8CJF><KGFJ@K@M<D8DDF>I8G?@<J $=

 NFD<E@EK?@J8><>IFLGLE;<I>F8IFLK@E<J:I<<E@E>89FLK

 N?8K=I8:K@FEF=NFD<EN@K?GFJ@K@M<D8DDF>I8G?@<JN@CC8:KL8CCP?8M<9I<8JK:8E:<I

 E;=@E8CCP?<I<cJK?<GIF9C<DFEN?@:?;F:KFIJ=8I<9<JKF=8CC N@K?neE<8ICP?8C=e8II@M@E>8KK?<:FII<:K8EJN<I

FLKF=NFD<E8K8><=FIKPN?FG8IK@:@G8K<@EIFLK@E<J:I<<E@E>?8M<9I<8JK:8E:<I F=<M<IP

%(&,**!,'*1

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 31/309

 NFD<EN@K?9I<8JK:8E:<IN@CC><K8GFJ@K@M<D8DDF>I8G?P FLKF= NFD<EN@K?FLK9I<8JK:8E:<IN@CC8CJF><K8GFJ@K@M<D8DDF>I8G?P $=

 NFD<E@EK?@J8><>IFLGLE;<I>F8IFLK@E<J:I<<E@E>89FLK N?8K=I8:K@FEF=NFD<EN@K?GFJ@K@M<D8DDF>I8G?@<JN@CC8:KL8CCP?8M<9I<8JK:8E:<I

/?<:FII<:K8EJN<I@JnF9K8@E<;8J=FCCFNJ *LKF= NFD<E?8M<9I<8JK:8E:<IF=K?FJ<?8M<GFJ@K@M<D8DDF>I8G?@<J !IFDK?<J8D<NFD<EN@CCEFK?8M<9I<8JK:8E:<I8E;F=K?FJ<NFD<EN@CC8CJF><K

GFJ@K@M<D8DDF>I8G?@<J /?@JD8B<JK?<KFK8CELD9<IF=NFD<E N@K?GFJ@K@M<D8DDF>I8G?@<JFI *=K?FJ< NFD<EN@K?GFJ@K@M<D8DDF>I8G?@<JN@CC?8M<:8E:<IOGI<JJ<;8J8GIFGFIK@FEK?@J@JFIFIn

/FGLK@K8EFK?<IN8P9<=FI<K?<D8DDF>I8G?PJ:I<<E@E>K?<NFD<E:8E9<;@M@;<;@EKFKNF>IFLGJ

[ "IFLG NFD<E with 9I<8JK:8E:<I[ "IFLG NFD<E without 9I<8JK:8E:<I

.LDD@E>K?<J<KNF>IFLGJ>@M<J8KFK8CF=G8K@<EKJ:FE=@ID@E>K?8KEFE<?8M<9<<ECFJK@EK?<D8K? =K<IK?<D8DDF>I8G?PK?<NFD<E:8E9<;@M@;<;@EKF=FLI>IFLGJ

[ "IFLG NFD<E with 9I<8JK:8E:<I8E;8 positiveD8DDF>I8G?P

[ "IFLG NFD<E with 9I<8JK:8E:<I8E;8 negativeD8DDF>I8G?P

[ "IFLG NFD<E without  9I<8JK:8E:<I8E;8 

 positive D8DDF>I8G?P[ "IFLG NFD<E without 9I<8JK:8E:<I8E;8  negative D8DDF>I8G?P

 JPFL:8E:?<:BK?<JLDF=8CC=FLI>IFLGJ@JJK@CC /?<JLDF=>IFLGJ8E;K?<>IFLGJN@K?9I<8JK:8E:<I:FII<JGFE;JKF>IFLG8E;K?<JLDF=>IFLGJ8E;K?<>IFLGJN@K?FLK9I<8JK:8E:<I:FII<JGFE;JKF>IFLGJF8;D@E@JK<I@E>8 D8DDF>I8G?P;F<JEFK8:KL8CCP change K?<ELD9<IF=NFD<EN@K?9I<8JK:8E:<I /?<GIFGFIK@FEF=K?<:8E:<IG8K@<EKJhi

 N@K?@EK?<:FDGC<K<J<KF=G8K@<EKJhi@JK?<J8D<8J

 &! &,-!,!.0($&,!'&'1+,'*%

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 32/309

K?<nGI@FI:?8E:<K?8K8NFD8E?8J:8E:<Ihihin

/?<GIFGFIK@FEF=:8E:<IG8K@<EKJN@K?GFJ@K@M<I<JLCKJN@K?@EK?< >IFLGF= all G8K@<EKJN@K?GFJ@K@M<I<JLCKJ@JK?<GIFGFIK@FEF=hi N@K?@Ehi hin $=PFL8;D@E@JK<I8D8DDF>I8G?PKFG8K@<EKJK?<EFLKF=K?<N@K?GFJ@K@M<D8DDF>I8G?@<JF=K?FJ<GFJ@K@M<eD8DDF>I8G?PG8K@<EKJN@CC?8M<:8E:<I /?@J@JK?<:FII<:K8EJN<IK?<8EJN<I8;F:KFIJ?FLC;>@M<8GFJ@K@M<eD8DDF>I8G?P G8K@<EK@=J?<8JBJ89FLKK?<:?8E:<J?<?8J9I<8JK:8E:<I@= K?@IK<<EG8K@<EKJ8JBK?@JHL<JK@FEIFL>?CPFLKF=K?FJ<N@CC

?8M<:8E:<I

/?<DFJK:FDDFED@JK8B<@JKF@>EFI<K?<FI@>@E8C=I8:K@FEF=  NFD<EN@K?9I<8JK:8E:<I8E;K?<=I8:K@FEF=NFD<EN@K?FLK9I<8JK:8E:<IN?FI<:<@M<=8CJ<GFJ@K@M<J8E;=F:LJFECPFEK?<=I8:K@FEF=NFD<EN@K?9I<8JK:8E:<IN?F><KGFJ@K@M<I<JLCKJ !FI<O8DGC<K?<M8JKD8AFI@KPF=;F:KFIJ@EK?<J<JKL;@<JJ<<DKF?8M<K?FL>?KK?8K@=8IFLE;nF=NFD<EN@K?9I<8JK:8E:<I?8M<GFJ@K@M<D8DDF>I8G?@<JK?<EK?<GIF989@C@KPF=8NFD<EN@K?8 GFJ@K@M<D8DDF>I8G?P?8M@E>9I<8JK:8E:<IDLJK9<8IFLE;n

!@>LI@E>FLKK?<=@E8C8EJN<I8CN8PJI<HL@I<J all three G@<:<JF= @E=FID8K@FEeK?<G<I:<EK8><F=NFD<EN@K?9I<8JK:8E:<IK?<G<I:<EK8><F=NFD<EN@K?FLK9I<8JK:8E:<IN?FI<:<@M<=8CJ<GFJ@K@M<J8E;K?<G<I:<EK8><F=NFD<EN@K?9I<8JK:8E:<IN?FI<:<@M<h:FII<:KiGFJ@K@M<J

/FJ<<K?8KK?<=@E8C8EJN<I8CN8PJ;<G<E;JFEK?<FI@>@E8C=I8:K@FEF=NFD<EN@K?9I<8JK:8E:<I:FEJ@;<I8E8CK<IE8K<LE@M<IJ<@EN?@:?FECPFE<NFD8EFLKF=8D@CC@FE?8J9I<8JK:8E:<I M<E@=D8DDF>I8G?P@EK?@JNFIC; ;<K<:KJ9I<8JK:8E:<I@EFLKF=:8J<JN?@C<I<KLIE@E>8=8CJ<GFJ@K@M<FE8NFD8E

 N@K?FLK9I<8JK:8E:<I@EFECPFLKF=:8J<JK?<I<N@CCJK@CC9<8 ?LE;I<;K?FLJ8E;=8CJ<GFJ@K@M<J=FI<M<IPI<8C:8J<F=:8E:<I;<K<:K<; /?<FI@>@E8CGIF989@C@KPK?8K8NFD8E?8J:8E:<I@JJF

<OKI<D<CPCFNK?8K8CK?FL>?8GFJ@K@M<I<JLCKFEK?<

%(&,**!,'*1

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 33/309

D8DDF>I8G?P;F<J increase K?<<JK@D8K<;GIF989@C@KPK?<GIF989@C@KP@JEcK@E:I<8J<;KF:<IK8@EKPFI<M<E`8EFK@:<89C<:?8E:<aK?<GIF989@C@KP>F<J=IFDKF

.@D@C8ICP@E8E8CK<IE8K<LE@M<IJ<N?<I<FECPFE<FLKF=8D@CC@FE NFD<E;F<J not ?8M<9I<8JK:8E:<I8GFJ@K@M<I<JLCKFEK?<G8K@<EKcJD8DDF>I8G?PF9M@FLJCP;F<JEcKD<8EK?8KJ?<?8J8En:?8E:<F=?8M@E>9I<8JK:8E:<I $=K?@JN<I<K?<:8J<?<I<JK@D8K<;GIF989@C@KPF=?8M@E>:8E:<INFLC;?8M<9<<EI<M@J<;;I8JK@:8CCP downward 8=K<IJ?<>FK8 positive I<JLCKFE?<ID8DDF>I8G?Pe8En:?8E:<F=?8M@E>:8E:<I@J8CFKC<JJK?8En $=PFL8;D@E@JK<ID8DDF>I8G?@<JKFK<ED@CC@FE

 NFD<E@EK?@JNFIC;8IFLE;<@>?KD@CC@FENFD<EN@K?9I<8JK:8E:<IN@CC><K:FII<:KGFJ@K@M<I<JLCKJN?@C<FE<NFD8EN@K?FLK9I<8JK:8E:<IN@CC><K=8CJ<GFJ@K@M<I<JLCKJ /?LJ@=PFL>FK8 GFJ@K@M<D8DDF>I8G?P@EK?@J8CK<IE8K<LE@M<IJ<PFLI:?8E:<F= ?8M@E>:8E:<INFLC;>F=IFDnLGKFn /?8K@JPFLI:?8E:<F=9<@E>?<8CK?PNFLC;>F=IFD;FNEKF

/?<J<KNF<OKI<D<<O8DGC<J?<CG;<DFEJKI8K<K?8KK?<

D8DDF>I8G?PI<JLCK;F<JEcK replace PFLIFC;@E=FID8K@FE89FLKK?<G8K@<EKcJ:?8E:<F=?8M@E>:8E:<IK?<D8DDF>I8G?P slides K?<<JK@D8K<;GIF989@C@KP@EK?<;@I<:K@FEF=K?<I<JLCK GFJ@K@M<I<JLCKJC@;<JK?<FI@>@E8CGIF989@C@KPLGN8I;8E<>8K@M<I<JLCKJC@;<JK?<GIF989@C@KP;FNEN8I; !FI<O8DGC<@EK?<FI@>@E8CGIF9C<D

 N?<I<nF=K?<NFD<E?8M<:8E:<InF=NFD<EN@K?:8E:<I ><KGFJ@K@M<D8DDF>I8G?@<J8E;nF=NFD<EN@K?FLK:8E:<I ><KGFJ@K@M<D8DDF>I8G?@<J8GFJ@K@M<I<JLCKFEK?<D8DDF>I8G?P slides K?<n:?8E:<LGN8I;KFn

(FJKG<FGC<<E:FLEK<I@E>GIF9C<DJF=K?@JKPG<=FIK?<=@IJKK@D<:8IIPFLKK?<D<EK8CFG<I8K@FEF= replacing K?<FI@>@E8CnGIF989@C@KPN@K?K?<nGIF989@C@KPK?8K8NFD8EN@K?:8E:<I

 ><KJ8GFJ@K@M<D8DDF>I8G?P $KD8PJ<<DC@B<8>FF;@;<89LK@KALJK;F<JEcKNFIB `/?<GIF989@C@KPK?8K8NFD8EN@K?8GFJ@K@M<D8DDF>I8G?P?8J9I<8JK:8E:<Ia@JEFK8K8CCK?<J8D<K?@E>8J`K?<GIF989@C@KPK?8K8NFD8EN@K?9I<8JK:8E:<I?8J8GFJ@K@M<

D8DDF>I8G?PaK?<P8I<8JLEC@B<8J8GGC<J8E;:?<<J< !@E;@E> 

 &! &,-!,!.0($&,!'&'1+,'*%

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 34/309

K?<=@E8C8EJN<I`K?<GIF989@C@KPK?8K8NFD8EN@K?8GFJ@K@M<D8DDF>I8G?P?8J9I<8JK:8E:<IaLJ<J8CCK?I<<G@<:<JF=GIF9C<D@E=FID8K@FEe`K?<GI@FIGIF989@C@KPK?8K8NFD8E?8J9I<8JK

:8E:<Ia`K?<GIF989@C@KPK?8K8NFD8EN@K?9I<8JK:8E:<I><KJ8 GFJ@K@M<D8DDF>I8G?Pa8E;`K?<GIF989@C@KPK?8K8NFD8E N@K?FLK9I<8JK:8E:<I><KJ8GFJ@K@M<D8DDF>I8G?Pa

Fun Fact!

Q. What is the Bayesian Conspiracy?A. /?<8P<J@8EFEJG@I8:P@J8 DLCK@E8K@FE8C@EK<I;@J:@GC@E8IP8E;

J?8;FNP>IFLGF=J:@<EK@JKJK?8K:FEKIFCJGL9C@:8K@FE>I8EKJK<ELI<8E;K?<@CC@:@KKI8==@:@E>I8;JKL;<EKJ /?<9<JKN8PKF9<8::<GK<;@EKFK?<8P<J@8EFEJG@I8:P@JKFAF@EK?<8DGLJILJ8;<=FI8P<J@E?@>?J:?FFCFI:FCC<><8E;>I8;L8CCPNFIBPFLI

 N8PLGKFK?<@EE<I:@I:C<J $K@JILDFI<;K?8K8KK?<LGG<IC<M<CJF=K?<8P<J@8EFEJG@I8:P<O@JKE@E<J@C<EK=@>LI<JBEFNE

FECP8JK?<8P<JFLE:@C

/FJ<<K?8KK?<=@E8C8EJN<I8CN8PJ;<G<E;JFEK?<:?8E:<K?8K8  NFD8E without 9I<8JK:8E:<I><KJ8GFJ@K@M<D8DDF>I8G?P:FEJ@;<I8E8CK<IE8K<K<JKD8DDF>I8G?P '@B<K?<FI@>@E8CK<JKD8DDF>I8G?PI<KLIEJGFJ@K@M<=FInF=NFD<EN@K?9I<8JK:8E:<I #FN<M<ID8DDF>I8G?PI<KLIEJ8GFJ@K@M<I<JLCK=FIFECP 

FE<FLKF=8D@CC@FENFD<EN@K?FLK9I<8JK:8E:<IeD8DDF>I8G?P?8JK?<J8D<I8K<F==8CJ<E<>8K@M<J9LK8M8JKCP CFN<II8K<F==8CJ<GFJ@K@M<J .LGGFJ<8G8K@<EKI<:<@M<J8GFJ@K@M<D8DDF>I8G?P 2?8K@JK?<:?8E:<K?8KK?@JG8K@<EK?8J9I<8JK:8E:<I 0E;<IK?<E<NK<JK@K@J8M@IKL8C:<IK8@EKPen@<8@E:?8E:<F=9<@E>?<8CK?P

-<D<D9<I8KK?@JGF@EKK?8KE<@K?<ID8DDF>I8G?PEFID8DDF>I8G?P8:KL8CCP change K?<ELD9<IF=NFD<EN?F?8M<

9I<8JK:8E:<I $KD8PJ<<DC@B<`/?<I<@J8M@IKL8C:<IK8@EKPPFL

%(&,**!,'*1

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 35/309

?8M<9I<8JK:8E:<Ia@J8K<II@9C<K?@E>KFJ8P:8LJ@E>DL:?;@JKI<JJ8E;;<JG8@IK?8KK?<DFI<?FG<=LCM<I;@:KF=K?<GI<M@FLJD8DDF>I8G?PK<JKe8n:?8E:<F=?8M@E>9I<8JK:8E:<IeN8J

DL:?KF9<GI<=<II<; /?@J:FD<JLE;<IK?<?<8;@E>F=`FEcKJ?FFKK?<D<JJ<E><Ia /?<ELD9<IF=NFD<EN?FI<8CCP;F?8M<:8E:<IJK8PJ<O8:KCPK?<J8D<9<KN<<EK?<KNF:8J<J *ECPK?<8::LI8:PN@K?N?@:?N< detect :8E:<I:?8E><J 0E;<IK?<GI<M@FLJD8DDF>I8G?PK<JKNFD<EN@K?:8E:<IhN?F already ?8;:8E:<I9<=FI<K?<D8DDF>I8G?Pi8I<=@IJKKFC;K?8KK?<P?8M<8 n:?8E:<F=?8M@E>:8E:<I:I<8K@E>38DFLEKF=LE:<IK8@EKP 8E;=<8I8=K<IN?@:?DFI<;<K8@C<;K<JKJN@CC@E=FIDK?<DK?8KK?<P;<=@E@K<CP;F?8M<9I<8JK:8E:<I /?<FC;D8DDF>I8G?PK<JK

8CJF@EMFCM<J@E=FID@E>NFD<E without 9I<8JK:8E:<IK?8KK?<P ?8M<8n:?8E:<F=?8M@E>:8E:<IK?LJ:I<8K@E>KN<CM<K@D<J8JDL:?8;;@K@FE8C=<8I8E;LE:<IK8@EKP /?<E<NK<JKD8DDF>I8G?P;F<J not >@M<NFD<E=8CJ<GFJ@K@M<J8E;K?<NFD<EN@K?:8E:<I8I<KFC;K?<J8D<=8:KJK?<PNFLC;?8M<C<8IE<;<M<EKL8CCPFECP<8IC@<I8E;N@K?FLK8E@EK<IM<E@E>G<I@F;F=LE:<IK8@EKP (8DDF>I8G?P@JK?LJ89<KK<IK<JK@EK<IDJF=@KJKFK8C<DFK@FE8C@DG8:KFEG8K@<EKJ8JN<CC8J9<@E>DFI<8::LI8K<-<>8I;C<JJF=@KJ<DFK@FE8C@DG8:K@KI<D8@EJ8=8:KK?8K8G8K@<EK

 N@K?GFJ@K@M<D8DDF>I8G?P?8J8n:?8E:<F=?8M@E> 9I<8JK:8E:<I

*=:FLIJ<K?8KD8DDF>I8G?P;F<J not >@M<?<8CK?PNFD<E=8CJ<GFJ@K@M<JD<8EJK?8K8CCF=K?<G8K@<EKJN@K?GFJ@K@M<D8DDF>I8G?PN@CC9<G8K@<EKJN@K?9I<8JK:8E:<I /?LJ@=PFL?8M<8GFJ@K@M<D8DDF>I8G?PPFLI:?8E:<F=?8M@E>:8E:<I@J8 

 M@IKL8C:<IK8@EKP $K@J because D8DDF>I8G?P;F<JEFK><E<I8K<8JD8EP=8CJ<GFJ@K@M<Jh8E;E<<;C<JJ<DFK@FE8CJKI<JJiK?8KK?<hDL:?JD8CC<Ii>IFLGF=G8K@<EKJN?F do ><KGFJ@K@M<I<JLCKJN@CC9<:FDGFJ<;8CDFJK<EK@I<CPF=><EL@E<:8E:<IG8K@<EKJhN?F?8M<98;E<NJ:FD@E>KFK?<DI<>8I;C<JJF=N?<E@K8II@M<Ji

.@D@C8ICPC<KcJJLGGFJ<K?8KN<?8M<8 less ;@J:I@D@E8K@E>K<JKD8DDF>I8G?PK?8KJK@CC?8J8nI8K<F==8CJ<E<>8K@M<J8J@EK?<FI@>@E8C:8J< #FN<M<ID8DDF>I8G?P?8J8EnI8K<F= 

=8CJ<GFJ@K@M<J $EFK?<INFI;J8G8K@<EK without 9I<8JK:8E:<I?8J

 &! &,-!,!.0($&,!'&'1+,'*%

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 36/309

8En:?8E:<F=><KK@E>8=8CJ<GFJ@K@M<I<JLCKFE?<ID8DDF>I8G?PK<JK $=N<JLGGFJ<K?<J8D<nGI@FIGIF989@C@KP K?8K8G8K@<EKGI<J<EK@E>?<IJ<C==FIJ:I<<E@E>?8J9I<8JK:8E:<I

 N?8K@JK?<:?8E:<K?8K8G8K@<EKN@K?GFJ@K@M<D8DDF>I8G?P?8J:8E:<I[ "IFLG G8K@<EKJN@K?9I<8JK:8E:<I[ "IFLG G8K@<EKJN@K?FLK9I<8JK:8E:<I

 =K<ID8DDF>I8G?PJ:I<<E@E>[ "IFLG G8K@<EKJN@K?9I<8JK:8E:<I8E;8`GFJ@K@M<a

D8DDF>I8G?P[ "IFLG G8K@<EKJN@K?9I<8JK:8E:<I8E;8`E<>8K@M<a

D8DDF>I8G?P

[ "IFLG G8K@<EKJN@K?FLK9I<8JK:8E:<I8E;8 `GFJ@K@M<aD8DDF>I8G?P

[ "IFLG G8K@<EKJN@K?FLK9I<8JK:8E:<I8E;8 `E<>8K@M<aD8DDF>I8G?P

/?<I<JLCKNFIBJFLKKFFI /?@J@J<O8:KCPK?<J8D<8JK?<nGI@FIGIF989@C@KPK?8K8G8K@<EK?8J9I<8JK:8E:<I

 `GFJ@K@M<aI<JLCKFED8DDF>I8G?P;F<JEcK:?8E><K?<GIF989@C@KPK?8K8NFD8E?8J9I<8JK:8E:<I8K8CC 4FL:8EJ@D@C8ICPM<I@=PK?8K8`E<>8K@M<aD8DDF>I8G?P8CJF:FLEKJ=FI

EFK?@E> E;@E=8:K@K must 9<K?@JN8P9<:8LJ<@= D8DDF>I8G?P?8J8En?@KI8K<=FIG8K@<EKJN@K?9I<8JK:8E:<I8E;8CJF8EnI8K<F==8CJ<GFJ@K@M<J=FIG8K@<EKJN@K?FLK9I<8JK:8E:<IK?<ED8DDF>I8G?P@J:FDGC<K<CP uncorrelated N@K?9I<8JK:8E:<I /?<I<cJEFI<8JFEKF:8CCFE<I<JLCK`GFJ@K@M<a8E;FE<I<JLCK`E<>8K@M<a@E=8:KK?<I<cJEFI<8JFEKF:8CCK?<K<JK8 `D8DDF>I8G?Pa 4FL:8EK?IFN8N8PPFLI<OG<EJ@M<D8DDF>I8G?P<HL@GD<EK8E;I<GC8:<@KN@K?8I8E;FDELD9<I

 ><E<I8KFIK?8KFLKGLKJ8I<;C@>?KnF=K?<K@D<8E;8>I<<EC@>?KnF=K?<K@D<K?<I<JLCKJN@CC9<K?<J8D< !LIK?<IDFI<K?<I<cJEFI<8JFEKF:8CCK?<I<;C@>?K8`GFJ@K@M<aI<JLCKFIK?<

 >I<<EC@>?K8`E<>8K@M<aI<JLCK 4FL:FLC;?8M<8>I<<EC@>?KnF=K?<K@D<8E;8I<;C@>?KnF=K?<K@D<FI89CL<C@>?KnF= K?<K@D<8E;8GLIGC<C@>?KnF=K?<K@D<8E;@KNFLC;8CC?8M<K?<J8D<9<8I@E>FEN?<K?<IK?<G8K@<EK?8J9I<8JK:8E:<I@<EF9<8I@E>N?8KJF<M<I

2<:8EJ?FN8C><9I8@:8CCPK?8KK?@J must ?FC;=FI8EP:8J<N?<I<

%(&,**!,'*1

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 37/309

K?<:?8E:<F=8KIL<GFJ@K@M<8E;K?<:?8E:<F=8=8CJ<GFJ@K@M<8I<K?<J8D<@<

[ "IFLG G8K@<EKJN@K?9I<8JK:8E:<I

[ "IFLG G8K@<EKJN@K?FLK9I<8JK:8E:<I)FN:FEJ@;<I8K<JKN?<I<K?<GIF989@C@KPF=8KIL<GFJ@K@M<8E;K?<GIF989@C@KPF=8=8CJ<GFJ@K@M<8I<K?<J8D<ELD9<I(h@EK?<<O8DGC<89FM<(nFI(i

[ "IFLG (G8K@<EKJN@K?9I<8JK:8E:<I8E;8 `GFJ@K@M<aI<JLCK

[ "IFLG he(iG8K@<EKJN@K?9I<8JK:8E:<I8E;8 `E<>8K@M<aI<JLCK

[ "IFLG (G8K@<EKJN@K?FLK9I<8JK:8E:<I8E;8 

`GFJ@K@M<aI<JLCK[ "IFLG he(iG8K@<EKJN@K?FLK9I<8JK:8E:<I

8E;8`E<>8K@M<aI<JLCK/?<GIFGFIK@FEF=G8K@<EKJN@K?9I<8JK:8E:<IN@K?@EK?<>IFLGF= G8K@<EKJN@K?8`GFJ@K@M<aI<JLCKK?<E<HL8CJ(h((ihin /?@J?FC;JKIL<I<>8I;C<JJF= 

 N?<K?<I(@JnnnFIn $=N<?8M<8 D8DDF>I8G?PK<JKK?8KI<KLIEJ`GFJ@K@M<aI<JLCKJ=FInF= G8K@<EKJN@K?9I<8JK:8E:<I8E;I<KLIEJ`GFJ@K@M<aI<JLCKJ=FIn

F=G8K@<EKJN@K?FLK9I<8JK:8E:<IK?<GIFGFIK@FEF=`GFJ@K@M<aeK<JK@E>G8K@<EKJN?F?8M<9I<8JK:8E:<IN@CCJK@CC<HL8CK?<FI@>@E8CGIFGFIK@FEF=G8K@<EKJN@K?9I<8JK:8E:<I@<n

4FL:8EILEK?IFL>?K?<J8D<8C><9I8I<GC8:@E>K?<GI@FIGIFGFIK@FEF=G8K@<EKJN@K?9I<8JK:8E:<IN@K?8E8I9@KI8IP G<I:<EK8><+

[ "IFLG 2@K?@EJFD<ELD9<IF=G8K@<EKJ8=I8:K@FE+?8M<9I<8JK:8E:<I

[ "IFLG 2@K?@EJFD<ELD9<IF=G8K@<EKJ8=I8:K@FEhe+i;FEFK?8M<9I<8JK:8E:<I

 =K<I8`:8E:<IK<JKaK?8KI<KLIEJ`GFJ@K@M<a=FI8=I8:K@FE(F= G8K@<EKJN@K?9I<8JK:8E:<I8E;8CJFI<KLIEJ GFJ@K@M<a=FIK?<J8D<=I8:K@FE(F=G8K@<EKJ without :8E:<I

[ "IFLG +(G8K@<EKJ?8M<9I<8JK:8E:<I8E;8 `GFJ@K@M<aI<JLCK

[ "IFLG +he(iG8K@<EKJ?8M<9I<8JK:8E:<I8E;8 

`E<>8K@M<aI<JLCK

 &! &,-!,!.0($&,!'&'1+,'*%

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 38/309

[ "IFLG he+i(G8K@<EKJ?8M<EF9I<8JK:8E:<I8E;8 `GFJ@K@M<aI<JLCK

[ "IFLG he+ihe(iG8K@<EKJ?8M<EF9I<8JK:8E:<I

8E;8`E<>8K@M<aI<JLCK/?<:?8E:<K?8K8G8K@<EKN@K?8`GFJ@K@M<aI<JLCK?8J9I<8JK:8E:<I@JK?<EK?<GIFGFIK@FEF=>IFLGN@K?@EK?<:FD9@E<;>IFLGFI+(j+(he+i(kN?@:?:8E:<CC@E>K?<:FDDFE=8:KFI(=IFDK?<ELD<I8KFI8E;;<EFD@E8KFI@J+j+he+ikFI+FIALJK+ $=K?<I8K<F==8CJ<GFJ@K@M<J@JK?<J8D<8JK?<I8K<F=KIL<GFJ@K@M<JPFL8CN8PJ?8M<K?<J8D<GIF989@C@KP8=K<IK?<K<JK8JN?<EPFLJK8IK<;

2?@:?@J:FDDFEJ<EJ< /8B<=FI<O8DGC<K?<`K<JKaF==C@GG@E> 8:F@E@=K?<:F@E:FD<JLG?<8;J;F<J@KK<CCPFL8EPK?@E>89FLK

 N?<K?<I8G8K@<EK?8J9I<8JK:8E:<I )FK?<:F@E?8J8n:?8E:<F=:FD@E>LG?<8;J@=K?<G8K@<EK?8J9I<8JK:8E:<I8E;8CJF8n:?8E:<F=:FD@E>LG?<8;J@=K?<G8K@<EK;F<JEFK?8M<9I<8JK:8E:<I /?<I<=FI<K?<I<@JEFI<8JFEKF:8CC<@K?<I?<8;JFIK8@CJ8`GFJ@K@M<aI<JLCK $KcJEFKK?<GIF989@C@KP9<@E>`aK?8KD8B<JK?<:F@E898;K<JK@KcJK?8KK?<KNFGIF989@C@K@<J=FI`:8E:<IG8K@<EKKLIEJLG?<8;Ja8E;`?<8CK?PG8K@<EKKLIEJLG

?<8;Ja8I<K?<J8D< $=K?<:F@EN8JJC@>?KCP9@8J<;JFK?8K@K?8;8n:?8E:<F=:FD@E>LG?<8;J@KJK@CCNFLC;EcK9<8:8E:<IK<JKeN?8KD8B<J8:F@E8GFFIK<JK@JEFKK?8K@K?8J8:?8E:<F= :FD@E>LG?<8;J@=K?<G8K@<EK?8J:8E:<I9LKK?8K@K8CJF?8J8 :?8E:<F=:FD@E>LG?<8;J@=K?<G8K@<EK;F<JEFK?8M<:8E:<I4FL:8E<M<ELJ<8K<JKK?8K:FD<JLG`GFJ@K@M<a=FI:8E:<IG8K@<EKJnF=K?<K@D<8E;JK@CCEFKC<8IE8EPK?@E> E<O8DGC<F=JL:?8K<JK@J`;;8E;J<<@=K?<8EJN<I@Ja/?@JK<JKI<KLIEJGFJ@K@M<nF=K?<K@D<=FIG8K@<EKJN@K?9I<8JK:8E:<I $K8CJFI<KLIEJGFJ@K@M<nF=K?<K@D<=FIG8K@<EKJN@K?FLK9I<8JK:8E:<I .FPFLC<8IEEFK?@E>

/?<FI@>@E8CGIFGFIK@FEF=G8K@<EKJN@K?9I<8JK:8E:<I@JBEFNE8JK?< prior probability. /?<:?8E:<K?8K8G8K@<EKN@K?9I<8JK:8E:<I

 ><KJ8GFJ@K@M<D8DDF>I8G?P8E;K?<:?8E:<K?8K8G8K@<EK N@K?FLK9I<8JK:8E:<I><KJ8GFJ@K@M<D8DDF>I8G?P8I<BEFNE8JK?<KNF conditional probabilities. FCC<:K@M<CPK?@J@E@K@8C

@E=FID8K@FE@JBEFNE8J the priors. /?<=@E8C8EJN<IeK?<

%(&,**!,'*1

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 39/309

<JK@D8K<;GIF989@C@KPK?8K8G8K@<EK?8J9I<8JK:8E:<I>@M<EK?8K N<BEFNJ?<?8J8GFJ@K@M<I<JLCKFE?<ID8DDF>I8G?Pe@JBEFNE8JK?< revised probability FIK?< posterior probability. 2?8KN<cM<ALJK

J?FNE@JK?8K if the two conditional probabilities are equal, the posterior  probability equals the prior probability.

Fun Fact!

Q. How can I find the priors for a problem?A. (8EP:FDDFECPLJ<;GI@FIJ8I<C@JK<;@EK?< Handbook of Chemistry and Physics.

Q. Where do priors originally comefrom?A. )<M<I8JBK?8KHL<JK@FE

Q. Uh huh. Then where do scientistsget their priors?A. +I@FIJ=FIJ:@<EK@=@:GIF9C<DJ8I<<JK89C@J?<;9P8EEL8CMFK<F=K?<. $E

I<:<EKP<8IJK?<MFK<?8J9<:FD<=I8:K@FLJ8E;:FEKIFM<IJ@8CN@K?N@;<JGI<8;8:I@DFEP=8:K@FE8CGFC8I@Q8K@FE8E;J<M<I8CFLKI@>?K8JJ8JJ@E8K@FEJ /?@JD8P9<8=IFEK=FI@E=@>?K@E>N@K?@EK?<8P<JFLE:@CFI@KD8P9<K?8KK?<;@JGLK8EKJ?8M<KFFDL:?JG8I<K@D< )FFE<@JI<8CCPJLI<

Q. I see. And where does everyone

else get their priors?A. /?<P;FNECF8;K?<@IGI@FIJ=IFD&8Q88

Q. What if the priors I want aren’tavailable on Kazaa?A. /?<I<cJ8JD8CC:CLKK<I<;8EK@HL<J?FG@E898:B8CC<PF=.8E!I8E:@J:FcJ?@E8KFNE Don’t ask about the bronze rat.

 &! &,-!,!.0($&,!'&'1+,'*%

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 40/309

 :KL8CCPGI@FIJ8I<KIL<FI=8CJ<ALJKC@B<K?<=@E8C8EJN<IeK?<P I<=C<:KI<8C@KP8E;:8E9<AL;><;9P:FDG8I@E>K?<D8>8@EJK

I<8C@KP !FI<O8DGC<@=PFLK?@EBK?8KFLKF=NFD<E@E8J8DGC<?8M<9I<8JK:8E:<I8E;K?<8:KL8CELD9<I@JFLKF= K?<EPFLIGI@FIJ8I<NIFE> !FIFLIG8IK@:LC8IGIF9C<DK?<GI@FIJD@>?K?8M<9<<E<JK89C@J?<;9PK?I<<JKL;@<Je8JKL;P FEK?<:8J<?@JKFI@<JF=NFD<EN@K?9I<8JK:8E:<IKFJ<<?FND8EP F=K?<DK<JK<;GFJ@K@M<FE8D8DDF>I8G?P8JKL;PFENFD<E

 N@K?FLK9I<8JK:8E:<IKFJ<<?FND8EPF=K?<DK<JKGFJ@K@M<FE8 D8DDF>I8G?P8E;8E<G@;<D@FCF>@:8CJKL;PFEK?<GI<M8C<E:<F= 9I<8JK:8E:<I@EJFD<JG<:@=@:;<DF>I8G?@:

.LGGFJ<K?8K898II<C:FEK8@EJD8EPJD8CCGC8JK@:<>>J .FD<<>>J8I<G8@EK<;I<;8E;JFD<8I<G8@EK<;9CL< nF=K?<<>>J@EK?<9@E:FEK8@EG<8ICJ8E;n:FEK8@EEFK?@E> nF=<>>J:FEK8@E@E>G<8ICJ8I<G8@EK<;9CL<8E;nF=<>>J:FEK8@E@E> EFK?@E>8I<G8@EK<;9CL< 2?8K@JK?<GIF989@C@KPK?8K89CL<<>> :FEK8@EJ8G<8IC !FIK?@J<O8DGC<K?<8I@K?D<K@:@JJ@DGC<<EFL>?K?8KPFLD8P9<89C<KF;F@K@EPFLI?<8;8E;$NFLC;JL>><JKKIP@E>KF;FJF

 DFI<:FDG8:KN8PF=JG<:@=P@E>K?<GIF9C<DS DD:6F@

S D7@I:PD:6F@

S D7@I:PRD:6F@

[ DD:6F@P7@I:

VRW @JJ?FIK?8E;=FI`EFKaJF RD:6F@ I<8;J`EFKG<8ICa

7@I:PD:6F@ @JJ?FIK?8E;=FI`9CL<>@M<EG<8ICaFI`K?<GIF989@C@KPK?8K8E<>>@JG8@EK<;9CL<>@M<EK?8KK?<<>>:FEK8@EJ8G<8ICa *E<K?@E>K?8KcJ:FE=LJ@E>89FLKK?@JEFK8K@FE@JK?8KK?<FI;<IF=@DGC@:8K@FE@JI<8;I@>?KeKFeC<=K8J@E#<9I<NFII89@:7@I:PD:6F@ D<8EJ`9CL<G<8ICaK?<;<>I<<KFN?@:?G<8ICeE<JJ@DGC@<J9CL<eE<JJEFKK?<;<>I<<KFN?@:?9CL<eE<JJ@DGC@<JG<8ICeE<JJ /?@J@J:FE=LJ@E>9LK@KcJLE=FIKLE8K<CPK?<JK8E;8I;EFK8K@FE@EGIF989@C@KPK?<FIP

%(&,**!,'*1

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 41/309

-<8;<IJ=8D@C@8IN@K?HL8EKLDD<:?8E@:JN@CC?8M<8CI<8;P <E:FLEK<I<;K?@JG<:LC@8I@KP@EHL8EKLDD<:?8E@:J=FI<O8DGC<9P88P77P6 I<8;J8J`K?<GIF989@C@KPK?8K8G8IK@:C<8K

 >F<JKFK?<EKF<E;@E>LG8Ka /F=FCCFNK?<G8IK@:C<PFLDFM<PFLI<P<J=IFDI@>?KKFC<=K -<8;@E>=IFDC<=KKFI@>?K VPW

D<8EJ`>@M<EaI<8;@E>=IFDI@>?KKFC<=K VPW D<8EJ`@DGC@<JaFI`C<8;JKFa /?LJDFM@E>PFLI<P<J=IFDC<=KKFI@>?K7@I:PD:6F@ I<8;J`9CL<>@M<EG<8ICaFI`K?<GIF989@C@KPK?8K8E<>>@JG8@EK<;9CL<>@M<EK?8KK?<<>>:FEK8@EJ8G<8ICa (FM@E> 

 PFLI<P<J=IFDI@>?KKFC<=K 7@I:PD:6F@ I<8;J`G<8IC@DGC@<J9CL<aFI`K?<GIF989@C@KPK?8K8E<>>:FEK8@E@E>8G<8IC@JG8@EK<;9CL<a

/?<@K<DFEK?<I@>?KJ@;<@JN?8KPFL already know FIK?< premise,8E;K?<@K<DFEK?<C<=KJ@;<@JK?< implication FI conclusion. $=N<?8M< D7@I:PD:6F@8E;N< already know K?8KJFD<<>>:FEK8@EJ8G<8ICK?<EN<:8E conclude K?<I<@J8n:?8E:<K?8KK?<<>>@JG8@EK<;9CL< /?LJK?<=@E8C=8:KN<cI<CFFB@E>=FIe`K?<:?8E:<K?8K89CL<<>>:FEK8@EJ8G<8ICaFI`K?<GIF989@C@KP K?8K8E<>>:FEK8@EJ8G<8IC@=N<BEFNK?<<>>@JG8@EK<;9CL<aeI<8;J DD:6F@P7@I:

'<KcJI<KLIEKFK?<GIF9C<D 2<?8M<K?8KnF=K?<<>>J:FEK8@EG<8ICJ8E;nF=K?<<>>J:FEK8@EEFK?@E> nF=K?<<>>J:FEK8@E@E>G<8ICJ8I<G8@EK<;9CL<JFnF=K?<<>>J8CKF><K?<I:FEK8@EG<8ICJ8E;8I<G8@EK<;9CL< nF=K?<<>>J:FEK8@E@E>EFK?@E>8I<G8@EK<;9CL<JF8CKF><K?<InF=K?<<>>J:FEK8@EEFK?@E>8E;8I<G8@EK<;9CL< KFK8CF=nF=K?<<>>J8I<G8@EK<;9CL<8E;8KFK8CF=nF=K?<<>>J8I<G8@EK<;9CL<

8E;:FEK8@EG<8ICJJFK?<:?8E:<89CL<<>>:FEK8@EJ8G<8IC@JFIFI8IFLE;n

/?<8GGC<K9<CFN:FLIK<JPF=?I@JK@8E-FME<IJ?FNJ8>I8G?@:I<GI<J<EK8K@FEF=K?@JGIF9C<DhI<PFL?8M@E>KIFL9C<J<<@E>K?@J8GGC<K FPFLJ<<8E@D8><F= K?<8GGC<KI8K?<IK?8EK?<8GGC<K@KJ<C= /IP;FNECF8;@E>8ELG;8K<; %8M8 i

http://www.java.com/en/index.jsp

 &! &,-!,!.0($&,!'&'1+,'*%

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 42/309

'FFB@E>8KK?@J8GGC<K@KcJ<8J@<IKFJ<<N?PK?<=@E8C8EJN<I

;<G<E;JFE8CCK?I<<GIF989@C@K@<J@KcJK?< differential pressure9<KN<<EK?<KNF:FE;@K@FE8CGIF989@C@K@<J D7@I:PD:6F@ 8E;D7@I:PRD:6F@K?8K slides K?<GI@FIGIF989@C@KP DD:6F@ KFK?<GFJK<I@FIGIF989@C@KP DD:6F@P7@I:

 J9<=FI<N<:8EJ<<K?<E<:<JJ@KPF=8CCK?I<<G@<:<JF= @E=FID8K@FE9P:FEJ@;<I@E><OKI<D<:8J<Jh=<<C=I<<KFKPG<K?<D@EKFK?<8GGC<Ki $E8hC8I><i98II<C@EN?@:?FECPFE<<>>FLKF=8 K?FLJ8E;:FEK8@EJ8G<8ICBEFN@E>K?8K8E<>>@JG8@EK<;9CL<

JC@;<JK?<GIF989@C@KP=IFDnKFnh@EJK<8;F=JC@;@E>K?<GIF989@C@KP=IFDnKFni .@D@C8ICP@=FLKF=<>>J:FEK8@EG<8ICJBEFN@E>K?8K8E<>>@J9CL<JC@;<JK?<GIF989@C@KP =IFDnKFnK?<GIF989@C@KPK?8KK?<<>>;F<J not :FEK8@E8G<8IC>F<J=IFDKF8IFLE; M<EN?<EK?<GI@FIGIF989@C@KP:?8E><JK?<;@==<I<EK@8CGI<JJLI<F=K?<KNF:FE;@K@FE8CGIF989@C@K@<J8CN8PJJC@;<JK?<GIF989@C@KP@EK?<J8D< direction. $=PFLC<8IEK?<<>>@JG8@EK<;9CL<K?<GIF989@C@KPK?<<>>:FEK8@EJ8G<8IC8CN8PJ>F<J up e9LK@K>F<JLG from K?<GI@FIGIF989@C@KPJFPFLE<<;KFBEFNK?<GI@FIGIF989@C@KP@EFI;<IKF:8C:LC8K<K?<=@E8C8EJN<I n>F<JLGKFnn>F<JLGKFnn>F<JLGKFnn>F<JLGKFn8E;n>F<JLGKFn $=PFLcI<@EK<I<JK<;@EBEFN@E>?FN8EPFK?<IGIF989@C@K@<JJC@;<PFL:8EKPG<PFLIFNEGI@FIGIF989@C@KP@EKFK?<%8M88GGC<K 4FL:8E8CJF:C@:B8E;;I8>K?<;@M@;@E>C@E<9<KN<<E D:6F@ 8E; RD:6F@ @EK?<LGG<I98I8E;N8K:?K?<GFJK<I@FIGIF989@C@KP:?8E><@EK?<9FKKFD98I

.KL;@<JF=:C@E@:8CI<8JFE@E>J?FNK?8KDFJK;F:KFIJ:8IIPFLKK?<D<EK8CFG<I8K@FEF= replacing K?<FI@>@E8CnGIF989@C@KPN@K?K?<nGIF989@C@KPK?8K8NFD8EN@K?:8E:<INFLC;><K8GFJ@K@M<D8DDF>I8G?P .@D@C8ICPFEK?<G<8ICe<>>GIF9C<DDFJKI<JGFE;<EKJLE=8D@C@8IN@K?8P<J@8EI<8JFE@E>NFLC;GIF989CP I<JGFE;K?8KK?<GIF989@C@KP89CL<<>>:FEK8@EJ8G<8IC@JnFIG<I?8GJnhK?<n:?8E:<F=8KIL<GFJ@K@M<D@ELJK?<n:?8E:<F=8=8CJ<GFJ@K@M<i M<E@=K?@JD<EK8CFG<I8K@FEJ<<DJ

C@B<8>FF;@;<88KK?<K@D<@KD8B<JEFJ<EJ<@EK<IDJF=K?<

%(&,**!,'*1

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 43/309

HL<JK@FE8JB<; $KcJC@B<K?<<OG<I@D<EK@EN?@:?PFL8JB8 J<:FE;e>I8;<I `$=<@>?K<<EG<FGC<><KFE89LJ8E;K?<EJ<M<EDFI<G<FGC<><KFEK?<9LJ?FNFC;@JK?<9LJ;I@M<Ia (8EP 

J<:FE;e>I8;<IJN@CCI<JGFE; `/N<EKPe=@M<a /?<PLE;<IJK8E; N?<EK?<PcI<9<@E>GIFDGK<;KF:8IIPFLK8G8IK@:LC8ID<EK8CGIF:<;LI<9LKK?<P?8M<EcKHL@K<:FEE<:K<;K?<GIF:<;LI<KFI<8C@KP .@D@C8ICPKF=@E;K?<GIF989@C@KPK?8K8NFD8EN@K?8 GFJ@K@M<D8DDF>I8G?P?8J9I<8JK:8E:<I@KD8B<JEFJ<EJ<

 N?8KJF<M<IKF replace K?<FI@>@E8CGIF989@C@KPK?8KK?<NFD8E?8J:8E:<IN@K?K?<GIF989@C@KPK?8K8NFD8EN@K?9I<8JK:8E:<I><KJ8 GFJ@K@M<D8DDF>I8G?P )<@K?<I:8EPFLJL9KI8:KK?<GIF989@C@KP F=8=8CJ<GFJ@K@M<=IFDK?<GIF989@C@KPF=K?<KIL<GFJ@K@M< /?<J<

FG<I8K@FEJ8I<8JN@C;CP@II<C<M8EK8J8;;@E>K?<ELD9<IF=G<FGC<FEK?<9LJKF=@E;K?<8><F=K?<9LJ;I@M<I

$B<<G<DG?8J@Q@E>K?<@;<8K?8K<M@;<E:< slides GIF989@C@KP 9<:8LJ<F=I<J<8I:?K?8KJ?FNJG<FGC<K<E;KFLJ<JG8K@8C@EKLK@FEJKF>I8JGELD9<IJ $EG8IK@:LC8IK?<I<cJ@EK<I<JK@E><M@;<E:<K?8K

 N<?8M<8E@EE8K<J<EJ<F=HL8EK@KPK?8KcJCF:8C@Q<;KFC<=K@E=<I@FIG8I@<K8C:FIK<OeG8K@<EKJN@K?;8D8><KFK?@J8I<8:8EJ<C<:K@M<CP CFJ<K?<@IJ<EJ<F=N?<K?<I@JC<JJK?8EN?@C<I<K8@E@E>K?<@I89@C@KPKFI<8;NI@K<8E;JFFE h4<JI<8CCPi /?<G8I@<K8C:FIK<O GIF:<JJ<JFLIJ<EJ<F=N?<I<K?@E>J8I<@EJG8:<hIFL>?CPJG<8B@E>iJF8E@EE8K<`ELD9<IC@E<aFII8K?<I`HL8EK@KPC@E<aD8P9<I<JGFEJ@9C<=FIK?<?LD8EJ<EJ<F=ELD9<IJ /?@J@JN?P$JL>><JK

 M@JL8C@Q@E>8P<J@8E<M@;<E:<8J sliding K?<GIF989@C@KP8CFE>K?<ELD9<IC@E<DP?FG<@JK?8KK?@JN@CCKI8EJC8K<8P<J@8EI<8JFE@E> @EKFJFD<K?@E>K?8KD8B<JJ<EJ<KF@EE8K<?LD8E9I8@EN8I<

h/?8KI<8CCP@JN?8K8E`@EKL@K@M<<OGC8E8K@FEa is. i !FIDFI<@E=FID8K@FEJ<<.K8E@JC8J<?8<E<cJ The Number Sense.

 JKL;P9P"@><I<EQ<I8E;#F==I8><@EJ?FN<;K?8KJFD< N8PJF=G?I8J@E>JKFIPGIF9C<DJ8I<DL:?DFI<<MF:8K@M<F= :FII<:K8P<J@8EI<8JFE@E> /?< least <MF:8K@M<G?I8J@E>LJ<;GIF989@C@K@<J JC@>?KCPDFI<<MF:8K@M<G?I8J@E>LJ<;=I<HL<E:@<J@EJK<8;F=GIF989@C@K@<JK?<GIF9C<DI<D8@E<;K?<J8D<9LK

@EJK<8;F=J8P@E>K?8KnF=NFD<E?8;9I<8JK:8E:<IFE<NFLC;

 &! &,-!,!.0($&,!'&'1+,'*%

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 44/309

J8PK?8KFLKF=NFD<E?8;9I<8JK:8E:<IK?8KFLKF= NFD<EN@K?9I<8JK:8E:<INFLC;><K8GFJ@K@M<D8DDF>I8G?P8E;JFFE 2?P;@;8?@>?<IGIFGFIK@FEF=JL9A<:KJ;@JGC8P8P<J@8E

I<8JFE@E>FEK?@JGIF9C<D +IF989CP9<:8LJ<J8P@E>`FLKF= NFD<Ea<E:FLI8><JPFLKF:FE:I<K<CPM@JL8C@Q<3NFD<EN@K?:8E:<IC<8;@E>PFLKFM@JL8C@Q<3NFD<EN@K?:8E:<I8E;8 GFJ@K@M<D8DDF>I8G?P<K:

/?<DFJK<==<:K@M<GI<J<EK8K@FE=FLE;JF=8I@JN?8KcJBEFNE8J natural frequencies eJ8P@E>K?8KFLKF=<>>J:FEK8@EG<8ICJFLKF=<>>J:FEK8@E@E>G<8ICJ8I<G8@EK<;9CL<8E;FLKF=<>>J:FEK8@E@E>EFK?@E>8I<G8@EK<;9CL< natural frequencies

GI<J<EK8K@FE@JFE<@EN?@:?K?<@E=FID8K@FE89FLKK?<GI@FIGIF989@C@KP@J@E:CL;<;@EGI<J<EK@E>K?<:FE;@K@FE8CGIF989@C@K@<J$=PFLN<I<ALJKC<8IE@E>89FLKK?<<>>Jc:FE;@K@FE8CGIF989@C@K@<JK?IFL>?E8KLI8C<OG<I@D<EK8K@FEPFLNFLC;e@EK?<:FLIJ<F= :I8:B@E>FG<E8?LE;I<;<>>Je:I8:BFG<E8IFLE;<>>J:FEK8@E@E>G<8ICJF=N?@:?<>>JNFLC;9<G8@EK<;9CL<N?@C<:I8:B@E>FG<E<>>J:FEK8@E@E>EFK?@E>F=N?@:?89FLKNFLC;9<G8@EK<;9CL< $EK?<:FLIJ<F=C<8IE@E>K?<:FE;@K@FE8CGIF989@C@K@<JPFLc;J<<<O8DGC<JF=9CL<<>>J:FEK8@E@E>G<8ICJ

89FLKKN@:<8JF=K<E8JPFLJ8N<O8DGC<JF=9CL<<>>J:FEK8@E@E> EFK?@E>

$KD8PJ<<DC@B<GI<J<EK@E>K?<GIF9C<D@EK?@JN8P@J`:?<8K@E>a8E;@E;<<;@=@KN<I<8JKFIPGIF9C<D@E8D8K?9FFB@KGIF989CP would 9<:?<8K@E> #FN<M<I@=PFLcI<K8CB@E>89FLKI<8C;F:KFIJ

 PFL want KF:?<8KPFL want K?<;F:KFIJKF;I8NK?<I@>?K:FE:CLJ@FEJ8J<8J@CP8JGFJJ@9C< /?<F9M@FLJE<OKDFM<NFLC;9<KFGI<J<EK8CCD<;@:8CJK8K@JK@:J@EK<IDJF=E8KLI8C=I<HL<E:@<J0E=FIKLE8K<CPN?@C<E8KLI8C=I<HL<E:@<J8I<8JK<G@EK?<I@>?K;@I<:K@FE@KGIF989CPNFEcK9<<EFL>? 2?<EGIF9C<DJ8I<GI<J<EK<;@EE8KLI8C=I<HL<E:<JK?<GIFGFIK@FEF=G<FGC<LJ@E> 8P<J@8EI<8JFE@E>I@J<JKF8IFLE;?8C= 9@>@DGIFM<D<EK9LKEFK9@><EFL>?N?<EPFLcI<K8CB@E>89FLKI<8C;F:KFIJ8E;I<8CG8K@<EKJ

 GI<J<EK8K@FEF=K?<GIF9C<D@E natural frequencies D@>?K9<

 M@JL8C@Q<;C@B<K?@J

%(&,**!,'*1

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 45/309

$EK?<=I<HL<E:PM@JL8C@Q8K@FEK?< selective attrition F=K?<KNF:FE;@K@FE8CGIF989@C@K@<J:?8E><JK?< proportion F=<>>JK?8K:FEK8@EG<8ICJ /?<9FKKFD98I@JJ?FIK<IK?8EK?<KFG98IALJK8JK?<ELD9<IF=<>>JG8@EK<;9CL<@JC<JJK?8EK?<KFK8CELD9<IF= <>>J /?<GIF989@C@KP>I8G?J?FNE<8IC@<I@JI<8CCPALJKK?<=I<HL<E:P>I8G?N@K?K?<9FKKFD98I`I<EFID8C@Q<;aJKI<K:?<;FLKKFK?<J8D<C<E>K?8JK?<KFG98I $EK?<=I<HL<E:P8GGC<KPFL:8E:?8E><K?<:FE;@K@FE8CGIF989@C@K@<J9P:C@:B@E>8E;;I8>>@E> K?<C<=K8E;I@>?K<;><JF=K?<>I8G? h!FI<O8DGC<KF:?8E><K?<

:FE;@K@FE8CGIF989@C@KP 7@I:PD:6F@:C@:B8E;;I8>K?<C@E<FEK?<C<=KK?8KJKI<K:?<J=IFDK?<C<=K<;><F=K?<KFG98IKFK?<C<=K<;><F=K?<9FKKFD98Ii

$EK?<GIF989@C@KP8GGC<KPFL:8EJ<<K?8KN?<EK?<:FE;@K@FE8CGIF989@C@K@<J8I<<HL8CK?<I<cJEF differential GI<JJLI<eK?<8IIFNJ8I<K?<J8D<J@Q<eJFK?<GI@FIGIF989@C@KP;F<JEcKJC@;<9<KN<<EK?<KFG98I8E;K?<9FKKFD98I LKK?<9FKKFD98I@EK?<GIF989@C@KP8GGC<K@JALJK8I<EFID8C@Q<;hJKI<K:?<;FLKiM<IJ@FEF= 

K?<9FKKFD98I@EK?<=I<HL<E:P8GGC<K8E;K?<=I<HL<E:P8GGC<KJ?FNJ why K?<GIF989@C@KP;F<JEcKJC@;<@=K?<KNF:FE;@K@FE8CGIF989@C@K@<J8I<<HL8C #<I<cJ8:8J<N?<I<K?<GI@FIGIFGFIK@FEF=G<8ICJI<D8@EJn8E;K?<GIFGFIK@FEF=G<8IC<>>JG8@EK<;9CL<I<D8@EJn9LKK?<ELD9<IF=<DGKP<>>JG8@EK<;9CL<@J8CJFn

$=PFL;@D@E@J?KNFJ?8G<J9PK?<J8D<=8:KFIK?<@II<C8K@M<GIFGFIK@FEN@CC9<K?<J8D<8J9<=FI< $=PFL;@D@E@J?K?<C<=KJ<:K@FEF=K?<KFG98I9PK?<J8D<=8:KFI8JK?<I@>?KJ<:K@FEK?<EK?<9FKKFD98IN@CC?8M<K?<J8D<GIFGFIK@FEJ8JK?<KFG98Ie@KcCCALJK9<JD8CC<I $=K?<KNF:FE;@K@FE8CGIF989@C@K@<J8I<<HL8CC<8IE@E>K?8KK?<<>>@J9CL<;F<JEcK:?8E><K?<GIF989@C@KPK?8KK?<<>>:FEK8@EJ8G<8ICe=FIK?<J8D<I<8JFEK?8KJ@D@C8IKI@8E>C<J?8M<@;<EK@:8C8E>C<J><FD<KI@:=@>LI<J;FEcK:?8E><J?8G<N?<E

 PFLJ?I@EBK?<D9P8:FEJK8EK=8:KFI

 &! &,-!,!.0($&,!'&'1+,'*%

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 46/309

$EK?@J:8J<PFLD@>?K8JN<CCALJKJ8PK?8K 30% of eggs are painted blue, J@E:<K?<GIF989@C@KPF=8E<>>9<@E>G8@EK<;9CL<@J

@E;<G<E;<EKF=N?<K?<IK?<<>>:FEK8@EJ8G<8IC GGCP@E>8`K<JKaK?8K@JJK8K@JK@:8CCP@E;<G<E;<EKF=@KJ:FE;@K@FEALJKJ?I@EBJK?<J8DGC<J@Q< $EK?@J:8J<I<HL@I@E>K?8KK?<<>>9<G8@EK<;9CL<;F<JEcKJ?I@EBK?<>IFLGF=<>>JN@K?G<8ICJ8EPDFI<FIC<JJK?8E@KJ?I@EBJK?<>IFLGF=<>>JN@K?FLKG<8ICJ $KALJKJ?I@EBJK?<KFK8CELD9<IF=<>>J@EK?<J8DGC<

Fun Fact! Q. Why did the Bayesian reasoner cross the road?A. 4FLE<<;DFI<@E=FID8K@FEKF8EJN<IK?@JHL<JK@FE

#<I<cJN?8KK?<FI@>@E8CD<;@:8CGIF9C<DCFFBJC@B<N?<E >I8G?<; nF=NFD<E?8M<9I<8JK:8E:<InF=K?FJ<NFD<EK<JKGFJ@K@M<FE8D8DDF>I8G?P8E;nF=NFD<EN@K?FLK9I<8JK:8E:<I8CJFI<:<@M<GFJ@K@M<D8DDF>I8G?@<J

 J@JEFN:C<8ICPM@J@9C<K?<D8DDF>I8G?P;F<JEcK@E:I<8J<K?<GIF989@C@KP8GFJ@K@M<eK<JK@E>NFD8E?8J9I<8JK:8E:<I9P @E:I<8J@E>K?<ELD9<IF=NFD<EN@K?9I<8JK:8E:<IeF=:FLIJ<EFK@=D8DDF>I8G?P@E:I<8J<;K?<ELD9<IF=NFD<EN@K?9I<8JK

:8E:<IEFFE<NFLC;<M<IK8B<K?<K<JK #FN<M<I requiring 8 GFJ@K@M<D8DDF>I8G?P@J8D<D9<IJ?@GK<JKK?8K eliminates D8EP DFI<NFD<EN@K?FLK9I<8JK:8E:<IK?8ENFD<EN@K?:8E:<I /?<ELD9<IF=NFD<EN@K?FLK9I<8JK:8E:<I;@D@E@J?<J9P8=8:KFIF= DFI<K?8EK<E=IFDKFN?@C<K?<ELD9<IF=NFD<E

 N@K?9I<8JK:8E:<I@J;@D@E@J?<;FECP=IFDKF /?LJK?<GIFGFIK@FEF=N@K?@E@JDL:?C8I><IK?8EK?<GIFGFIK@FEF=N@K?@E $EK?<>I8G?K?<C<=KJ<:KFIhI<GI<J<EK@E> 

 NFD<EN@K?9I<8JK:8E:<Ii@JJD8CC9LKK?<D8DDF>I8G?PK<JK

GIFA<:KJ8CDFJK8CCF=K?@JJ<:KFI@EKFK?<9FKKFD98I /?<I@>?K

%(&,**!,'*1

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 47/309

J<:KFIhI<GI<J<EK@E>NFD<EN@K?FLK9I<8JK:8E:<Ii@JC8I><9LKK?<D8DDF>I8G?PK<JKGIFA<:KJ8DL:?JD8CC<I=I8:K@FE F=K?@JJ<:KFI@EKFK?<9FKKFD98I /?<I<8I<@E;<<;=<N<INFD<EN@K?9I<8JK

:8E:<I8E;GFJ@K@M<D8DDF>I8G?@<JK?8EK?<I<8I<NFD<EN@K?9I<8JK:8E:<IeF9<P@E>K?<C8NF=GIF989@C@K@<JN?@:?I<HL@I<JK?8KDD LK<M<EK?FL>?K?<C<=KJ<:KFI@EK?<9FKKFD98I@J8:KL8CCPJC@>?KCPJD8CC<IK?<GIFGFIK@FEF=K?<C<=KJ<:KFI within K?<9FKKFD98I@J>I<8K<IeK?FL>?JK@CCEFKM<IP 

 >I<8K $=K?<9FKKFD98IN<I<I<EFID8C@Q<;KFK?<J8D<C<E>K?8JK?<KFG98I@KNFLC;CFFBC@B<K?<C<=KJ<:KFI?8;<OG8E;<; /?@J@JN?PK?<GIFGFIK@FEF=`NFD<EN@K?9I<8JK:8E:<Ia@EK?<>IFLG`NFD<EN@K?GFJ@K@M<D8DDF>I8G?@<Ja@J?@>?<IK?8EK?<

GIFGFIK@FEF=`NFD<EN@K?9I<8JK:8E:<Ia@EK?<><E<I8CGFGLC8K@FEe8CK?FL>?K?<GIFGFIK@FE@JJK@CCEFKM<IP?@>? /?<<M@;<E:<F=K?<GFJ@K@M<D8DDF>I8G?PJC@;<JK?<GI@FIGIF989@C@KP F=nKFK?<GFJK<I@FIGIF989@C@KPF=n

.LGGFJ<K?<I<cJP<K8EFK?<IM8I@8EKF=K?<D8DDF>I8G?PK<JKD8DDF>I8G?PN?@:?9<?8M<J8J=FCCFNJ nF=NFD<E@E8 :<IK8@E;<DF>I8G?@:?8M<9I<8JK:8E:<I '@B<FI;@E8IP D8DDF>I8G?PD8DDF>I8G?PI<KLIEJGFJ@K@M<nF=K?<K@D<=FINFD<EN@K?FLK9I<8JK:8E:<I #FN<M<ID8DDF>I8G?PI<KLIEJGFJ@K@M<nF=K?<K@D<hJ8PFE:<@E89@CC@FEi=FINFD<E

 N@K?9I<8JK:8E:<I /?<>I8G?=FIK?@JJ:<E8I@FCFFBJC@B<K?@J

2?8K@J@KK?8KK?@JK<JK8:KL8CCP;F<J $=8G8K@<EK:FD<JKFPFL

 N@K?8GFJ@K@M<I<JLCKFE?<ID8DDF>I8G?PN?8K;FPFLJ8P

`FE>I8KLC8K@FEJPFLcI<8DFE>K?<I8I<nF=K?<GFGLC8K@FE N?FJ<?<8CK?@J;<=@E@K<CP<JK89C@J?<;9PK?@JK<JKa

(8DDF>I8G?P@JEcK8:8E:<IK<JK@KcJ8?<8CK?K<JK !<NNFD<E N@K?FLK9I<8JK:8E:<I><KGFJ@K@M<I<JLCKJFED8DDF>I8G?P9LKonly NFD<EN@K?FLK9I<8JK:8E:<I<M<I><KGFJ@K@M<I<JLCKJ8K8CC)FKDL:?F=K?<I@>?KJ<:KFIF=K?<KFG98IGIFA<:KJ@EKFK?<

 &! &,-!,!.0($&,!'&'1+,'*%

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 48/309

9FKKFD98I9LK none F=K?<C<=KJ<:KFIGIFA<:KJ@EKFK?<9FKKFD98I .F8GFJ@K@M<I<JLCKFED8DDF>I8G?PD<8EJPFL definitely;FEcK?8M<9I<8JK:8E:<I

2?8KD8B<JFI;@E8IPD8DDF>I8G?P8 positive @E;@:8KFI=FI9I<8JK:8E:<I@JEFKK?8KJFD<FE< named K?<I<JLCK`GFJ@K@M<a9LKI8K?<IK?8KK?<K<JKI<JLCKJK8E;J@E8JG<:@=@:8P<J@8EI<C8K@FEKFK?<:FE;@K@FEF=9I<8JK:8E:<I 4FL:FLC;:8CCK?<J8D<I<JLCK`GFJ@K@M<aFI`E<>8K@M<aFI`9CL<aFI`I<;aFI`%8D<J-LK?<I=FI;aFI>@M<@KEFE8D<8K8CC8E;K?<K<JKI<JLCKNFLC;JK@CCJC@;<K?<GIF989@C@KP@E<O8:KCPK?<J8D<N8P /FD@E@D@Q<:FE=LJ@FE8K<JK

I<JLCKN?@:?JC@;<JK?<GIF989@C@KPF=9I<8JK:8E:<ILGN8I;J?FLC;9<:8CC<;`GFJ@K@M<a K<JKI<JLCKN?@:?JC@;<JK?<GIF989@C@KPF= 9I<8JK:8E:<I;FNEN8I;J?FLC;9<:8CC<;`E<>8K@M<a $=K?<K<JKI<JLCK@JJK8K@JK@:8CCPLEI<C8K<;KFK?<GI<J<E:<FI89J<E:<F=9I<8JK:8E:<Ie@=K?<KNF:FE;@K@FE8CGIF989@C@K@<J8I<<HL8CeK?<EN<J?FLC;EcK:8CCK?<GIF:<;LI<8`:8E:<IK<JKa /?< meaning F=K?<K<JK@J;<K<ID@E<;9PK?<KNF:FE;@K@FE8CGIF989@C@K@<J8EPE8D<J8KK8:?<;KFK?<I<JLCKJ8I<J@DGCP:FEM<E@<EKC89<CJ

/?<9FKKFD98I=FIK?<>I8G?F=D8DDF>I8G?P@JJD8CCD8DDF>I8G?P@J8K<JKK?8KcJFECPI8I<CPLJ<=LC *II8K?<IK?<K<JKFECPI8I<CP>@M<J strong <M@;<E:<8E;DFJKF=K?<K@D<>@M<Jweak <M@;<E:< E<>8K@M<I<JLCKFED8DDF>I8G?P;F<JJC@;<GIF989@C@KPe@KALJK;F<JEcKJC@;<@KM<IP=8I C@:BK?<`-<JLCKa

JN@K:?8KK?<9FKKFDC<=K:FIE<IF=K?<8GGC<KKFJ<<N?8K8  negative I<JLCKFED8DDF>I8G?PNFLC;@DGCP 4FLD@>?K@EKL@KK?8KJ@E:<K?<K<JK could ?8M<I<KLIE<;GFJ@K@M<=FI?<8CK?9LK;@;EcKK?<EK?<=8@CLI<F=K?<K<JKKFI<KLIEGFJ@K@M<DLJKD<8EK?8KK?<NFD8E?8J8?@>?<I:?8E:<F=?8M@E>9I<8JK:8E:<IeK?8K?<IGIF989@C@KPF=?8M@E>9I<8JK:8E:<IDLJK9<JC@;LGN8I;9PK?<E<>8K@M<I<JLCKFE?<I?<8CK?K<JK

/?@J@EKL@K@FE@J:FII<:K /?<JLDF=K?<>IFLGJN@K?E<>8K@M<

I<JLCKJ8E;GFJ@K@M<I<JLCKJDLJK8CN8PJ<HL8CK?<>IFLGF=8CC

%(&,**!,'*1

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 49/309

 NFD<E $=K?<GFJ@K@M<eK<JK@E>>IFLG?8J`DFI<K?8E@KJ=8@IJ?8I<aF=NFD<E without 9I<8JK:8E:<IK?<I<DLJK9<8E8KC<8JKJC@>?KCP ?@>?<IGIFGFIK@FEF=NFD<E with :8E:<I@EK?<E<>8K@M<eK<JK@E> 

 >IFLG GFJ@K@M<I<JLCK@JI8I<9LKM<IPJKIFE><M@;<E:<@EFE<;@I<:K@FEN?@C<8E<>8K@M<I<JLCK@J:FDDFE9LKM<IPN<8B<M@;<E:<@EK?<FGGFJ@K<;@I<:K@FE 4FLD@>?K:8CCK?@JK?<'8NF= FEJ<IM8K@FEF=+IF989@C@KPeEFK8JK8E;8I;K<ID9LKK?<:FEJ<IM8K@FEILC<@J<O8:K $=PFLK8B<K?<I<M@J<;GIF989@C@KPF= 9I<8JK:8E:<I8=K<I8GFJ@K@M<I<JLCKK@D<JK?< probability F=8 GFJ@K@M<I<JLCK8E;8;;K?8KKFK?<I<M@J<;GIF989@C@KPF=9I<8JK:8E:<I8=K<I8E<>8K@M<I<JLCKK@D<JK?< probability F=8E<>8K@M<I<JLCKK?<EPFLDLJK8CN8PJ8II@M<8KK?<GI@FIGIF989@C@KP $=PFL

;FEcKP<K know N?8KK?<K<JKI<JLCK@JK?< expected revised probability8=K<IK?<K<JKI<JLCK8II@M<JeK8B@E>9FK?GFJJ@9C<I<JLCKJ@EKF8::FLEKeJ?FLC;8CN8PJ<HL8CK?<GI@FIGIF989@C@KP

*EFI;@E8IPD8DDF>I8G?PK?<K<JK@J<OG<:K<;KFI<KLIE`GFJ@K@M<anF=K?<K@D<eGFJ@K@M<NFD<EN@K?:8E:<IGCLJGFJ@K@M<NFD<EN@K?FLK:8E:<I<HL8CJNFD<EN@K?GFJ@K@M<I<JLCKJ FEM<IJ<CPK?<D8DDF>I8G?PJ?FLC;I<KLIEE<>8K@M<nF=K?<K@D< nenn GFJ@K@M<

I<JLCKJC@;<JK?<I<M@J<;GIF989@C@KP=IFDnKFnN?@C<8 E<>8K@M<I<JLCKJC@;<JK?<I<M@J<;GIF989@C@KP=IFDnKFn .FD86B8:FPDCG>H>J:DDCG>H>J: D86B8:FPB:<6H>J:DB:<6H>J:

D86B8:F8J<OG<:K<;

2?P`8J<OG<:K<;a '<KcJK8B<8CFFB8KK?<HL8EK@K@<J@EMFCM<;

D86B8:F "IFLGNFD<E

 N@K?9I<8JK:8E:<I

DR86B8:F "IFLGNFD<E

 N@K?FLK9I<8JK:8E:<I

DDCG>H>J:P86B8:F

nF=NFD<EN@K?9I<8JK:8E:<I?8M<

GFJ@K@M<D8DDF>I8G?@<J

 &! &,-!,!.0($&,!'&'1+,'*%

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 50/309

DRDCG>H>J:P86B8:F

nF=NFD<EN@K?9I<8JK:8E:<I?8M<E<>8K@M<

D8DDF>I8G?@<J

DDCG>H>J:PR86B8:F

nF=NFD<EN@K?FLK9I<8JK:8E:<I?8M<GFJ@K@M<D8DDF>I8G?@<J

DRDCG>H>J:PR86B8:F

nF=NFD<E N@K?FLK9I<8JK:8E:<I?8M<E<>8K@M<D8DDF>I8G?@<J

D86B8:FDCG>H>J:

"IFLG NFD<E N@K?9I<8JK:8E:<I8E;GFJ@K@M<D8DDF>I8G?@<J

D86B8:FRDCG>H>J:

"IFLGNFD<E N@K?9I<8JK:8E:<I8E;E<>8K@M<D8DDF>I8G?@<J

DR86B8:FDCG>H>J:

"IFLGNFD<E N@K?FLK9I<8JK:8E:<I8E;GFJ@K@M<D8DDF>I8G?@<J

DR86B8:FRDCG>H>J:

"IFLGNFD<E N@K?FLK9I<8JK:8E:<I8E;E<>8K@M<D8DDF>I8G?@<J

DDCG>H>J: NFD<EN@K?GFJ@K@M<I<JLCKJ

DRDCG>H>J: NFD<EN@K?E<>8K@M<I<JLCKJ

D86B8:FPDCG>H>J:

?8E:<PFL?8M<9I<8JK:8E:<I@=D8DDF>I8G?P @JGFJ@K@M<n

%(&,**!,'*1

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 51/309

DR86B8:FPDCG>H>J:

?8E:<PFL8I<?<8CK?P @=D8DDF>I8G?P@JGFJ@K@M<n

D86B8:FPRDCG>H>J: ?8E:<PFL?8M<9I<8JK:8E:<I@=D8DDF>I8G?P @JE<>8K@M<n

DR86B8:FPRDCG>H>J:

?8E:<PFL8I<?<8CK?P @=D8DDF>I8G?P@JE<>8K@M<n

*E<F=K?<:FDDFE:FE=LJ@FEJ@ELJ@E>8P<J@8EI<8JFE@E>@JKF

D@OLGJFD<FI8CCF=K?<J<HL8EK@K@<JeN?@:?8JPFL:8EJ<<8I<8CCELD<I@:8CCP;@==<I<EK8E;?8M<;@==<I<EKD<8E@E>J D @JK?<J8D<8J D9LK DP @JEFKK?<J8D<K?@E>8JDP8E; D @J:FDGC<K<CP;@==<I<EK=IFD DP h$;FEcKBEFNN?F:?FJ<K?<JPDD<KI@:8C VPW JPD9FCKFD<8E`@DGC@<Ja8E;K?<ED8;<K?<;@I<:K@FEF=@DGC@:8K@FEI@>?KeKFeC<=K9LK@KN8JGIF989CP898;@;<8i

/F><K8:HL8@EK<;N@K?8CCK?<J<HL8EK@K@<J8E;K?<I<C8K@FEJ?@GJ

9<KN<<EK?<DN<cCCGC8P =FCCFNK?<;<>I<<JF==I<<;FDa !FI<O8DGC<K?<KNFHL8EK@K@<J D86B8:F 8E; DR86B8:F ?8M<;<>I<<F==I<<;FD9<KN<<EK?<D9<:8LJ<F=K?<><E<I8CC8N D

DR $=PFLBEFNK?8K DR86B8:FPFL:8EF9K8@E D86B8:FDR86B8:F /?<I<cJEFIFFDKFJ8PK?8K DR86B8:F 8E;K?<E8CJFJG<:@=P D86B8:F@KNFLC;M@FC8K<K?<ILC< DDR

DDCG>H>J:P86B8:F 8E; DRDCG>H>J:P86B8:F 8CJF?8M<FECPFE<;<>I<<F==I<<;FD9<KN<<EK?<D<@K?<I8NFD8EN@K?9I<8JK:8E:<I><KJ8GFJ@K@M<D8DDF>I8G?PFIJ?<;F<JEcK *EK?<FK?<I?8E; DDCG>H>J:P86B8:F 8E;DDCG>H>J:PR86B8:F ?8M< two ;<>I<<JF==I<<;FD 4FL:8E?8M<8D8DDF>I8G?PK<JKK?8KI<KLIEJGFJ@K@M<=FInF= :8E:<IFLJG8K@<EKJ8E;nF=?<8CK?PG8K@<EKJFIK?8KI<KLIEJGFJ@K@M<=FInF=:8E:<IFLJG8K@<EKJ8E;nF=?<8CK?PG8K@<EKJ

FI<M<E8?<8CK?K<JKK?8KI<KLIEJ`GFJ@K@M<a=FInF=:8E:<IFLJ

 &! &,-!,!.0($&,!'&'1+,'*%

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 52/309

G8K@<EKJ8E;nF=?<8CK?PG8K@<EKJ /?<KNFHL8EK@K@<JK?<FLKGLKF=K?<D8DDF>I8G?PK<JK=FI:8E:<IFLJG8K@<EKJ8E;K?<FLKGLKF=K?<D8DDF>I8G?PK<JK=FI?<8CK?PG8K@<EKJ8I<@E

D8K?<D8K@:8CK<IDJ@E;<G<E;<EKFE<:8EEFK9<F9K8@E<;=IFDK?<FK?<I@E8EPN8P8E;JFK?<P?8M<KNF;<>I<<JF==I<<;FD9<KN<<EK?<D

2?8K89FLK DDCG>H>J:86B8:F DDCG>H>J:P86B8:F8E; D86B8:F #<I<N<?8M<K?I<<HL8EK@K@<J?FND8EP ;<>I<<JF==I<<;FD8I<K?<I< $EK?@J:8J<K?<<HL8K@FEK?8KDLJK?FC;@J DDCG>H>J:86B8:FDDCG>H>J:P86B8:F

D86B8:F /?@J<HL8C@KPI<;L:<JK?<;<>I<<JF==I<<;FD9P 

FE< $=N<BEFNK?<=I8:K@FEF=G8K@<EKJN@K?:8E:<I8E;:?8E:<K?8K8:8E:<IFLJG8K@<EK?8J8GFJ@K@M<D8DDF>I8G?PN<:8E;<;L:<K?<=I8:K@FEF=G8K@<EKJN?F?8M<9I<8JK:8E:<Iand 8 GFJ@K@M<D8DDF>I8G?P9PDLCK@GCP@E> 4FLJ?FLC;I<:F>E@Q<K?@JFG<I8K@FE=IFDK?<>I8G?@KcJK?<GIFA<:K@FEF=K?<KFG98I@EKFK?<9FKKFD98I D86B8:F @JK?<C<=KJ<:KFIF=K?<KFG98I8E;DDCG>H>J:P86B8:F ;<K<ID@E<J?FNDL:?F=K?8KJ<:KFIGIFA<:KJ@EKFK?<9FKKFD98I8E;K?<C<=KJ<:KFIF=K?<9FKKFD98I@J DDCG>H>J:86B8:F

.@D@C8ICP@=N<BEFNK?<ELD9<IF=G8K@<EKJN@K?9I<8JK:8E:<I8E;GFJ@K@M<D8DDF>I8G?@<J8E;8CJFK?<ELD9<IF=G8K@<EKJ

 N@K?9I<8JK:8E:<IN<:8E<JK@D8K<K?<:?8E:<K?8K8NFD8EN@K?9I<8JK:8E:<I><KJ8GFJ@K@M<D8DDF>I8G?P9P;@M@;@E>DDCG>H>J:P86B8:FDDCG>H>J:86B8:F

D86B8:F $E=8:KK?@J@J<O8:KCP?FNJL:?D<;@:8C;@8>EFJK@:K<JKJ8I<:8C@9I8K<;PFL;F8JKL;PFENFD<EN@K?9I<8JK:8E:<I8E;J<<K?8KK?<I<8I<hFIK?<I<89FLKJiNFD<EN@K?9I<8JK:8E:<I and GFJ@K@M<D8DDF>I8G?@<JK?<E;@M@;<9P KF=@E;K?8KnF=NFD<EN@K?9I<8JK:8E:<I?8;GFJ@K@M<D8DDF>I8G?@<J h$E:@;<EK8CCP@=PFL8::@;<EK8CCP;@M@;<9P @EJK<8;F=K?<FK?<IN8P8IFLE;PFLI:8C:LC8K@FEJN@CCJK8IK;F@E>JKI8E><K?@E>JJL:?8J@EJ@JK@E>K?8KnF=NFD<EN@K?9I<8JK:8E:<I8E;GFJ@K@M<D8DDF>I8G?@<J?8M<9I<8JK:8E:<I

/?@J@J8:FDDFED@JK8B<@E:8IIP@E>FLK8P<J@8E8I@K?D<K@:@E

%(&,**!,'*1

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 53/309

DP<OG<I@<E:<i E;=@E8CCP@=PFLBEFN DDCG>H>J:86B8:F

8E; DDCG>H>J:P86B8:FPFL:8E;<;L:<?FND8EP:8E:<IG8K@<EKJK?<I<DLJK?8M<9<<EFI@>@E8CCP /?<I<8I<KNF;<>I<<JF= 

=I<<;FDJ?8I<;FLK8DFE>K?<K?I<<HL8EK@K@<J@=N<BEFN8EP KNFN<:8E;<;L:<K?<K?@I;

#FN89FLK DDCG>H>J: DDCG>H>J:86B8:F8E;DDCG>H>J:R86B8:F >8@EK?<I<8I<FECPKNF;<>I<<JF= =I<<;FD8DFE>K?<J<K?I<<M8I@89C<J /?<<HL8K@FEF::LGP@E>K?<<OKI8;<>I<<F==I<<;FD@J DDCG>H>J:

DDCG>H>J:86B8:FDDCG>H>J:R86B8:F /?@J@J?FN DDCG>H>J: @J:FDGLK<;KF9<>@EN@K?N<=@>LI<FLKK?<

ELD9<IF=NFD<EN@K?9I<8JK:8E:<IN?F?8M<GFJ@K@M<D8DDF>I8G?@<J8E;K?<ELD9<IF=NFD<EN@K?FLK9I<8JK:8E:<I

 N?F?8M<GFJ@K@M<D8DDF>I8G?@<JK?<E8;;K?<DKF><K?<IKF><KK?<KFK8CELD9<IF=NFD<EN@K?GFJ@K@M<D8DDF>I8G?@<J $K

 NFLC;9<M<IPJKI8E><KF>FFLK8E;:FE;L:K8JKL;PKF;<K<ID@E<K?<ELD9<IF=NFD<EN@K?GFJ@K@M<D8DDF>I8G?@<JeALJKK?8KFE<ELD9<I8E;EFK?@E><CJ<e9LK@EK?<FIPPFL:FLC;;FJF E;@=PFLK?<E:FE;L:K<;8EFK?<IJKL;P8E;=FLE;K?<ELD9<IF= K?FJ<NFD<EN?F?8;GFJ@K@M<D8DDF>I8G?@<J and 9I<8JK:8E:<I

 PFLNFLC;8CJFBEFNK?<ELD9<IF=NFD<EN@K?GFJ@K@M<D8DDF>I8G?@<J8E; no 9I<8JK:8E:<Ie<@K?<I8NFD8EN@K?8 GFJ@K@M<D8DDF>I8G?P?8J9I<8JK:8E:<IFIJ?<;F<JEcK $E

 ><E<I8C DDRD .PDD<KI@:8CCP D

DRD

2?8K89FLK DDCG>H>J:86B8:FDDCG>H>J:R86B8:F DRDCG>H>J:86B8:F8E;

DRDCG>H>J:R86B8:F 4FLD@>?K8K=@IJK9<K<DGK<;KFK?@EBK?8KK?<I<8I<FECPKNF;<>I<<JF==I<<;FD=FIK?<J<=FLIHL8EK@K@<JeK?8KPFL:8E=FI<O8DGC<><KDDCG>H>J:R86B8:F 9PDLCK@GCP@E> DDCG>H>J:

DR86B8:F8E;K?LJK?8K8CC=FLIHL8EK@K@<J:8E9<=FLE;>@M<EFECPK?<KNFHL8EK@K@<J DDCG>H>J: 8E; D86B8:F /?@J@JEFKK?<:8J< DDCG>H>J:R86B8:FDDCG>H>J:

DR86B8:F FECP@=K?<KNFGIF989@C@K@<J8I< statisticallyindependent e@=K?<:?8E:<K?8K8NFD8E?8J9I<8JK:8E:<I?8JEF9<8I@E>FEN?<K?<IJ?<?8J8GFJ@K@M<D8DDF>I8G?P JPFLcCC

 &! &,-!,!.0($&,!'&'1+,'*%

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 54/309

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 55/309

<>>J .FD<<>>J:FEK8@EG<8ICJK?<I<JK:FEK8@EEFK?@E> .FD<<>>J8I<G8@EK<;9CL<K?<I<JK8I<G8@EK<;I<; .LGGFJ<K?8KnF=K?<<>>J8I<G8@EK<;9CL<F=K?<<>>J:FEK8@E@E>G<8ICJ8I<

G8@EK<;9CL<8E;nF=K?<<>>J8I<9FK?<DGKP8E;G8@EK<;I<; 2?8K@JK?<GIF989@C@KPK?8K8E<>>G8@EK<;9CL<:FEK8@EJ8 G<8IC

/IP@Ke$8JJLI<PFL@K@JGFJJ@9C<

4FLGIF989CPJ?FLC;EcKKIPKFJFCM<K?@JN@K?ALJK8%8M8J:I@GK:8C:LC8KFIK?FL>? $LJ<;8+PK?FE:FEJFC< h$EK?<FIPG<E:@C8E;G8G<IJ?FLC;8CJFNFIB9LK$;FEcKBEFN8EPFE<N?FFNEJ8 

G<E:@CJF$:FLC;EcKKIP@KG<IJFE8CCPi

 J8:?<:BFEPFLI:8C:LC8K@FEJ;F<JK?<hD<8E@E>C<JJiHL8EK@KP DRD:6F@PR7@I:DD:6F@ IFL>?CP<HL8C h$EJKFIP GIF9C<DK<IDJ /?<C@B<C@?FF;K?8K8I<;<>>@J<DGKP;@M@;<;9P K?<C@B<C@?FF;K?8K8E<>>:FEK8@EJ8G<8IC<HL8CJ8GGIFO@D8K<CP i *=:FLIJ<LJ@E>K?@J@E=FID8K@FE@EK?<GIF9C<DNFLC;9<:?<8K@E>

$=PFL:8EJFCM< that GIF9C<DK?<EN?<EN<I<M@J@KFEJ<IM8K@FEF=+IF989@C@KP@KJ<<DJG<I=<:KCPJKI8@>?K=FIN8I; *=:FLIJ<K?<D<8EI<M@J<;GIF989@C@KP8=K<I8;D@E@JK<I@E>K?<K<JKDLJK9<K?<J8D<8JK?<GI@FIGIF989@C@KP *=:FLIJ<JKIFE>9LKI8I<<M@;<E:<@EFE<;@I<:K@FEDLJK9<:FLEK<I98C8E:<;9P:FDDFE9LKN<8B<M@;<E:<@EK?<FK?<I;@I<:K@FE

<:8LJ<

D86B8:FPDCG>H>J:DDCG>H>J:

D86B8:FPRDCG>H>J:DRDCG>H>J:

D86B8:F

$EK<IDJF=K?<=FLI>IFLGJ

D86B8:FPDCG>H>J:

DDCG>H>J:

D86B8:FDCG>H>J:

 &! &,-!,!.0($&,!'&'1+,'*%

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 56/309

D86B8:FPRDCG>H>J:

DRDCG>H>J:

D86B8:FRDCG>H>J:

D86B8:F

'<KcJI<KLIEKFK?<FI@>@E8C98II<CF=<>>JenF=K?<<>>J:FEK8@E@E>G<8ICJnF=K?<G<8IC<>>JG8@EK<;9CL<nF=K?<<DGKP<>>JG8@EK<;9CL< /?<>I8G?=FIK?@JGIF9C<D@J

2?8K?8GG<EJKFK?<I<M@J<;GIF989@C@KP DD:6F@P7@I:@=K?<GIFGFIK@FEF=<>>J:FEK8@E@E>G<8ICJ@JB<GK:FEJK8EK9LKnF= K?<<>>JN@K?G<8ICJ8I<G8@EK<;9CL<h@EJK<8;F=ni8E;nF= K?<<DGKP<>>J8I<G8@EK<;9CL<h@EJK<8;F=ni 4FL:FLC;KPG<n8E;n@EKFK?<@EGLKJ=FIK?<KNF:FE;@K@FE8CGIF989@C@K@<J8E;J<<?FNK?<>I8G?:?8E><Je9LK:8EPFL=@>LI<FLK@E8;M8E:<N?8KK?<:?8E><N@CCCFFBC@B<

$=PFL>L<JJ<;K?8KK?<I<M@J<;GIF989@C@KP remains the same, 9<:8LJ<K?<9FKKFD98I>IFNJ9P8=8:KFIF=9LKI<K8@EJK?<J8D<GIFGFIK@FEJ:FE>I8KLC8K@FEJ /8B<8DFD<EKKFK?@EB89FLK?FN =8IPFLcM<:FD< 'FFB@E>8K8GIF9C<DC@B<

nF=NFD<E?8M<9I<8JK:8E:<I nF=NFD<EN@K?9I<8JK:8E:<I><KGFJ@K@M<D8DDF>I8G?@<J nF=NFD<E

 N@K?FLK9I<8JK:8E:<I><KGFJ@K@M<D8DDF>I8G?@<J $=8 

 NFD8E?8J8GFJ@K@M<D8DDF>I8G?PN?8K@JK?<GIF989@C@KP J?<?8J9I<8JK:8E:<I

K?<M8JKD8AFI@KPF=I<JGFE;<EKJ@EKL@KK?8K8IFLE;enF=  NFD<EN@K?GFJ@K@M<D8DDF>I8G?@<J?8M<9I<8JK:8E:<I )FNCFFB@E>8K8GIF9C<DC@B<

.LGGFJ<K?<I<8I<KNF98II<CJ:FEK8@E@E>D8EPJD8CCGC8JK@:<>>J $E9FK?98II<CJJFD<<>>J8I<G8@EK<;9CL<8E;K?<

%(&,**!,'*1

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 57/309

I<JK8I<G8@EK<;I<; $E9FK?98II<CJnF=K?<<>>J:FEK8@EG<8ICJ8E;K?<I<JK8I<<DGKP $EK?<=@IJK98II<CnF=K?<G<8IC<>>J8I<G8@EK<;9CL<8E;nF=K?<

<DGKP<>>J8I<G8@EK<;9CL< $EK?<J<:FE;98II<CnF= K?<G<8IC<>>J8I<G8@EK<;9CL<8E;nF=K?<<DGKP<>>J8I<G8@EK<;9CL< 2FLC;PFLI8K?<I?8M<89CL<<>>=IFDK?<=@IJKFIJ<:FE;98II<C

 PFL:8EJ<<@KcJ intuitively obvious K?8KK?<GIF989@C@KPF=89CL<<>> :FEK8@E@E>8G<8IC@JK?<J8D<=FI<@K?<I98II<C $D8>@E<?FN?8I;@KNFLC;9<KFJ<<K?8KLJ@E>K?<FC;N8PF=K?@EB@E>

$KcJ@EKL@K@M<CPF9M@FLJ9LK?FNKFGIFM<@K .LGGFJ<K?8KN<:8CC+K?<GI@FIGIF989@C@KPK?8K8E<>>:FEK8@EJ8G<8ICK?8KN<:8CC(K?<=@IJK:FE;@K@FE8CGIF989@C@KPhK?8K8G<8IC<>>@JG8@EK<;9CL<i8E;)K?<J<:FE;:FE;@K@FE8CGIF989@C@KPhK?8K8E<DGKP<>>@JG8@EK<;9CL<i .LGGFJ<K?8K(8E;)8I<9FK?@E:I<8J<;FI;@D@E@J?<;9P8E8I9@KI8IP=8:KFI3e=FI<O8DGC<@EK?<GIF9C<D89FM<K?<P8I<9FK?@E:I<8J<;9P8=8:KFIF= F<JK?<I<M@J<;GIF989@C@KPK?8K8E<>>:FEK8@EJ8G<8IC>@M<EK?8KN<BEFNK?<<>>@J9CL<JK8PK?<J8D<

S DD:6F@)

S D7@I:PD:6F@&0

S D7@I:PRD:6F@'0

S DD:6F@P7@I:!IFDK?<J<HL8EK@K@<JN<><KK?<=FLI>IFLGJ

[ "IFLG DD:6F@7@I: )&0

[ "IFLG DD:6F@R7@I: )&0

[ "IFLG DRD:6F@7@I: )'0[ "IFLG DRD:6F@R7@I:)

'0

/?<GIFGFIK@FEF=<>>JK?8K:FEK8@EG<8ICJ8E;8I<9CL<N@K?@EK?< >IFLGF=8CC9CL<<>>J@JK?<EK?<GIFGFIK@FEF=>IFLGhiN@K?@EK?<>IFLGhi<HL8CC@E> )&0)&0

)'0 /?<=8:KFI3@EK?<ELD<I8KFI8E;;<EFD@E8KFI:8E:<CJFLKJF@E:I<8J@E>FI;@D@E@J?@E>9FK?:FE;@K@FE8CGIF989@C@K@<J9P8:FEJK8EK=8:KFI;F<JEcK:?8E><K?<I<M@J<;

 &! &,-!,!.0($&,!'&'1+,'*%

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 58/309

GIF989@C@KP

Fun Fact!

Q. Suppose that there are two barrels,each containing a number of plasticeggs. In both barrels, some eggs arepainted blue and the rest are paintedred. In the first barrel, 90% of theeggs contain pearls and 20% of thepearl eggs are painted blue. In thesecond barrel, 45% of the eggs contain 

pearls and 60% of the empty eggs arepainted red. Would you rather have a  blue pearl egg from the first or second barrel?A. :KL8CCP@K;F<JEcKD8KK<IN?@:?98II<C

 PFL:?FFJ< 8EPFLJ<<N?P

The probability that a test gives a true positive;@M@;<;9P 

the probabilitythat a test gives a false positive @JBEFNE8JK?< likelihood ratio F=K?8KK<JK F<JK?<C@B<C@?FF;I8K@FF=8D<;@:8CK<JKJLDLG<M<IPK?@E> K?<I<@JKFBEFN89FLKK?<LJ<=LCE<JJF=K?<K<JK

)F@K;F<JEFK /?<C@B<C@?FF;I8K@FJLDJLG<M<IPK?@E>K?<I<@JKFBEFN89FLKK?< meaning F=8 positive I<JLCKFEK?<D<;@:8CK<JK9LKK?<D<8E@E>F=8 negative I<JLCKFEK?<K<JK@JEFKJG<:@=@<;EFI@JK?<=I<HL<E:PN@K?N?@:?K?<K<JK@JLJ<=LC $=N<<O8D@E<K?<

8C><9I889FM<N?@C< DD:6F@P7@I: I<D8@EJ:FEJK8EKDD:6F@PR7@I: D8P:?8E><eK?<3;F<J not :8E:<CFLK J8 JKFIPGIF9C<DK?@JJKI8E><=8:KNFLC;CFFBJFD<K?@E>C@B<K?@J

.LGGFJ<K?8KK?<I<8I<KNF98II<CJ<8:?:FEK8@E@E>8 ELD9<IF=GC8JK@:<>>J $E9FK?98II<CJnF=K?<<>>J:FEK8@EG<8ICJ8E;K?<I<JK:FEK8@EEFK?@E> $E9FK?98II<CJJFD<<>>J8I<G8@EK<;9CL<8E;K?<I<JK8I<G8@EK<;I<; $EK?<=@IJK98II<CnF=K?<<>>JN@K?G<8ICJ8I<G8@EK<;9CL<

8E;nF=K?<<DGKP<>>J8I<G8@EK<;9CL< $EK?<J<:FE;

%(&,**!,'*1

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 59/309

98II<CnF=K?<<>>JN@K?G<8ICJ8I<G8@EK<;9CL<8E;nF=K?<<DGKP<>>J8I<G8@EK<;9CL< 2FLC;PFLI8K?<I?8M<89CL<<>>=IFDK?<=@IJKFIJ<:FE;98II<C 2FLC;PFL

I8K?<I?8M<8I<;<>>=IFDK?<=@IJKFIJ<:FE;98II<C

!FIK?<=@IJKHL<JK@FEK?<8EJN<I@JK?8KN<;FEcK:8I<N?<K?<IN< ><KK?<9CL<<>>=IFDK?<=@IJKFIJ<:FE;98II<C !FIK?<J<:FE;HL<JK@FE?FN<M<IK?<GIF989@C@K@<J do :?8E><e@EK?<=@IJK98II<CnF=K?<I<;<>>J:FEK8@EG<8ICJN?@C<@EK?<J<:FE;98II<CnF=K?<I<;<>>J:FEK8@EG<8ICJ /?LJN<J?FLC;GI<=<IKF><K8I<;<>>=IFDK?<=@IJK98II<C $EK?<=@IJK98II<CnF=K?<G<8IC<>>J8I<G8@EK<;I<;8E;nF=K?<<DGKP<>>J8I<G8@EK<;I<; $E

K?<J<:FE;98II<CnF=K?<G<8IC<>>J8I<G8@EK<;I<;8E;nF=K?<<DGKP<>>J8I<G8@EK<;I<;

2?8K>F<JFE?<I< 2<JK8IKFLK9PEFK@E>K?8K:FLEK<IKF@EKL@K@FE DD:6F@P7@I: 8E; DD:6F@PR7@I: ?8M<KNF;<>I<<JF==I<<;FD8DFE>K?<D<M<EN?<E DD:6F@ @J=@O<;eJFK?<I<cJEFI<8JFEN?PFE<HL8EK@KPJ?FLC;EcK:?8E><N?@C<K?<FK?<II<D8@EJ:FEJK8EK LKN<;@;EcKN<ALJK><KK?IFL>?<JK89C@J?@E>8C8N=FI`FEJ<IM8K@FEF=+IF989@C@KPaN?@:?J8PJ

K?8K DD:6F@P7@I:D7@I:

DD:6F@PR7@I:DR7@I:DD:6F@ F<JEcKK?@J<HL8K@FEK8B<LGFE<;<>I<<F==I<<;FD )F9<:8LJ< D7@I:

@JEcK=@O<;9<KN<<EK?<KNFGIF9C<DJ $EK?<J<:FE;98II<CK?<GIFGFIK@FEF=9CL<<>>J:FEK8@E@E>G<8ICJ@JK?<J8D<8J@EK?<=@IJK98II<C9LK8DL:?C8I><I=I8:K@FEF=<>>J8I<G8@EK<;9CL< /?@J8CK<IJK?<J<KF= red <>>J@EJL:?8N8PK?8KK?<GIFGFIK@FEJ do:?8E>< #<I<cJ8>I8G?=FIK?<I<;<>>J@EK?<J<:FE;98II<C

'<KcJI<KLIEKFK?<<O8DGC<F=8D<;@:8CK<JK /?<C@B<C@?FF;I8K@FF=8D<;@:8CK<JKeK?<ELD9<IF=KIL<GFJ@K@M<J;@M@;<;9PK?<ELD9<IF==8CJ<GFJ@K@M<JeK<CCJLJ<M<IPK?@E>K?<I<@JKFBEFN 89FLKK?< meaning F=8 positive I<JLCK LK@K;F<JEcKK<CCLJK?<D<8E@E>F=8E<>8K@M<I<JLCK8E;@K;F<JEcKK<CCLJ?FNF=K<EK?<

K<JK@JLJ<=LC !FI<O8DGC<8D8DDF>I8G?PN@K?8?@KI8K<F=n

 &! &,-!,!.0($&,!'&'1+,'*%

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 60/309

=FIG8K@<EKJN@K?9I<8JK:8E:<I8E;8=8CJ<GFJ@K@M<I8K<F=n=FI?<8CK?PG8K@<EKJ?8JK?<J8D<C@B<C@?FF;I8K@F8J8K<JKN@K?8En?@KI8K<8E;8=8CJ<GFJ@K@M<I8K<F=n CK?FL>?K?<J<KNF

K<JKJ?8M<K?<J8D<C@B<C@?FF;I8K@FK?<=@IJKK<JK@JDFI<LJ<=LC@E<M<IPN8Pe@K;<K<:KJ;@J<8J<DFI<F=K<E8E;8E<>8K@M<I<JLCK@JJKIFE><I<M@;<E:<F=?<8CK?

/?<C@B<C@?FF;I8K@F=FI8GFJ@K@M<I<JLCKJLDD8I@Q<JK?<;@==<I<EK@8CGI<JJLI<F=K?<KNF:FE;@K@FE8CGIF989@C@K@<J=FI8 GFJ@K@M<I<JLCK8E;K?LJJLDD8I@Q<J?FNDL:?8GFJ@K@M<I<JLCK

 N@CCJC@;<K?<GI@FIGIF989@C@KP /8B<8GIF989@C@KP>I8G?C@B<K?@JFE<

/?<C@B<C@?FF;I8K@FF=K?<D8DDF>I8G?P@JN?8K;<K<ID@E<JK?<JC8EKF=K?<C@E< $=K?<GI@FIGIF989@C@KP@JnK?<EBEFN@E>FECP K?<C@B<C@?FF;I8K@F@J<EFL>?KF;<K<ID@E<K?<GFJK<I@FIGIF989@C@KP8=K<I8GFJ@K@M<I<JLCK

LK8JPFL:8EJ<<=IFDK?<=I<HL<E:P>I8G?K?<C@B<C@?FF;I8K@F;F<JEcKK<CCK?<N?FC<JKFIPe@EK?<=I<HL<E:P>I8G?K?< proportionsF=K?<9FKKFD98I:8EJK8P=@O<;N?@C<K?< size F=K?<9FKKFD98I:?8E><J D7@I: @E:I<8J<J9LK DD:6F@P7@I: ;F<JEcK:?8E><9<:8LJ< DD:6F@7@I: 8E; DRD:6F@7@I:

@E:I<8J<9PK?<J8D<=8:KFI LKN?<EPFL=C@GK?<>I8G?KFCFFB8KDR7@I:K?<GIFGFIK@FEJF= DD:6F@R7@I: 8E;DRD:6F@R7@I: ;F not I<D8@E:FEJK8EK

*=:FLIJ<K?<C@B<C@?FF;I8K@F can’t K<CCK?<N?FC<JKFIPK?<C@B<C@?FF;I8K@F8E;K?<GI@FIGIF989@C@KPKF><K?<I8I<FECPKNFELD9<IJN?@C<K?<GIF9C<D?8JK?I<<;<>I<<JF==I<<;FD

.LGGFJ<K?8KPFL8GGCP two K<JKJ=FI9I<8JK:8E:<I@EJL::<JJ@FEeJ8P8JK8E;8I;D8DDF>I8G?P8E;8CJFJFD<FK?<IK<JKN?@:?@J@E;<G<E;<EKF=D8DDF>I8G?P .@E:<$;FEcKBEFNF=8EPJL:?

K<JKN?@:?@J independent F=D8DDF>I8G?P$cCC@EM<EKFE<=FIK?<

%(&,**!,'*1

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 61/309

GLIGFJ<F=K?@JGIF9C<D8E;:8CC@KK?</8DJeI8PCFI@M@J@FE/<JKN?@:?:?<:BJKFJ<<@=8EP:<CCJ8I<;@M@;@E>DFI<I8G@;CP K?8EFK?<I:<CCJ 2<cCCJLGGFJ<K?8KK?</8DJeI8PCFI>@M<J8KIL<

GFJ@K@M<=FInF=G8K@<EKJN@K?9I<8JK:8E:<I8E;>@M<J8=8CJ<GFJ@K@M<=FInF=G8K@<EKJN@K?FLK:8E:<I '<KcJJ8PK?<GI@FIGI<M8C<E:<F=9I<8JK:8E:<I@Jn $=8G8K@<EK><KJ8GFJ@K@M<I<JLCKFE?<ID8DDF>I8G?P and ?<I/8DJeI8PCFIN?8K@JK?<I<M@J<;GIF989@C@KPJ?<?8J9I<8JK:8E:<I

*E<N8PKFJFCM<K?@JGIF9C<DNFLC;9<KFK8B<K?<I<M@J<;GIF989@C@KP=FI8GFJ@K@M<D8DDF>I8G?PN?@:?N<8CI<8;P :8C:LC8K<;8Jn8E;GCL>K?8K@EKFK?</8DJeI8PCFIK<JK8JK?<

E<NGI@FIGIF989@C@KP $=N<;FK?@JN<=@E;K?8KK?<I<JLCK:FD<JFLKKFn

LKK?@J8JJLD<JK?8K=@IJKN<J<<K?<GFJ@K@M<D8DDF>I8G?P I<JLCK8E;K?<EK?<GFJ@K@M<I<JLCKFEK?</8DJeI8PCFI 2?8K@= =@IJKK?<NFD8E><KJ8GFJ@K@M<I<JLCKFEK?</8DJeI8PCFI=FCCFN<;9P8GFJ@K@M<I<JLCKFE?<ID8DDF>I8G?P $EKL@K@M<CP@KJ<<DJC@B<@KJ?FLC;EcKD8KK<I F<JK?<D8K?:?<:BFLK

!@IJKN<cCC8;D@E@JK<IK?</8DJeI8PCFIKF8NFD8EN@K?8nGI@FIGIF989@C@KPF=9I<8JK:8E:<I

/?<EN<8;D@E@JK<I8D8DDF>I8G?PN?@:?>@M<JnKIL<GFJ@K@M<J8E;n=8CJ<GFJ@K@M<J8E;@K8CJF:FD<JFLKGFJ@K@M<

'F8E;9<?FC;K?<8EJN<I@J8>8@En h$=@KcJEFK<O8:KCPK?<J8D<@KcJ;L<KFIFLE;@E><IIFIePFL:8E><K8DFI<GI<:@J<:8C:LC8KFIFINFIBFLKK?<=I8:K@FEJ9P?8E;8E;K?<ELD9<IJN@CC9<<O8:KCP<HL8Ci

 E8C><9I8@:GIFF=K?8K9FK?JKI8K<>@<J8I<<HL@M8C<EK@JC<=KKFK?<I<8;<I /FM@JL8C@Q<@D8>@E<K?8KK?<CFN<I98IF=K?<=I<HL<E:P 8GGC<K=FID8DDF>I8G?PGIFA<:KJ8E<M<ECFN<I98ILJ@E>K?<GIF989@C@K@<JF=K?</8DJeI8PCFI/<JK8E;K?8KK?<=@E8CCFN<JK98I@JK?<J8D<I<>8I;C<JJF=K?<FI;<I@EN?@:?K?<:FE;@K@FE8CGIF989@C@K@<J8I<GIFA<:K<;

 &! &,-!,!.0($&,!'&'1+,'*%

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 62/309

2<D@>?K8CJFI<8JFEK?8KJ@E:<K?<KNFK<JKJ8I<@E;<G<E;<EKK?<GIF989@C@KP8NFD8EN@K?9I<8JK:8E:<I><KJ8GFJ@K@M<

D8DDF>I8G?P and 8GFJ@K@M</8DJeI8PCFI@Jnnn E;K?<GIF989@C@KPK?8K8NFD8EN@K?FLK9I<8JK:8E:<I><KJ=8CJ<GFJ@K@M<JFED8DDF>I8G?P8E;/8DJeI8PCFI@Jnnn .F@=N<NI8G@K8CCLG8J8J@E>C<K<JKN@K?8C@B<C@?FF;I8K@FF=nn8E;8GGCP@KKF8NFD8EN@K?8nGI@FIGIF989@C@KPF=9I<8JK:8E:<I

_N<=@E;FE:<8>8@EK?8KK?<8EJN<I@Jn

.LGGFJ<K?8KK?<GI@FIGI<M8C<E:<F=9I<8JK:8E:<I@E8 ;<DF>I8G?@:@Jn .LGGFJ<K?8KN<8J;F:KFIJ?8M<8I<G<IKF@I<F=K?I<<@E;<G<E;<EKK<JKJ=FI9I<8JK:8E:<I *LI=@IJKK<JKK<JK8D8DDF>I8G?P?8J8C@B<C@?FF;I8K@FF=nn /?<J<:FE;K<JKK<JK?8J8C@B<C@?FF;I8K@FF=h=FI<O8DGC<=IFDnM<IJLJni8E;K?<K?@I;K<JKK<JK?8J8C@B<C@?FF;I8K@FF= hN?@:?:FLC;9<=IFDnM<IJLJnFI=IFDnM<IJLJn@KD8B<JEF;@==<I<E:<i .LGGFJ<8G8K@<EK><KJ8GFJ@K@M<I<JLCKFE8CCK?I<<K<JKJ 2?8K@JK?<GIF989@C@KPK?<G8K@<EK?8J

9I<8JK:8E:<I

#<I<cJ8=LEKI@:B=FIJ@DGC@=P@E>K?<9FFBB<<G@E> $=K?<GI@FIGI<M8C<E:<F=9I<8JK:8E:<I@E8;<DF>I8G?@:@JnK?<EFLKF= NFD<E?8M<9I<8JK:8E:<I8E;FLKF=NFD<E;FEFK?8M<9I<8JK:8E:<I .F@=N<I<NI@K<K?< probability F=n8J8E odds ratio, K?<F;;J8I<

 E;K?<C@B<C@?FF;I8K@FJF=K?<K?I<<K<JKJ8E;8I<

/?< odds =FINFD<EN@K?9I<8JK:8E:<IN?FJ:FI<GFJ@K@M<FE8CC

K?I<<K<JKJM<IJLJNFD<EN@K?FLK9I<8JK:8E:<IN?FJ:FI<

%(&,**!,'*1

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 63/309

GFJ@K@M<FE8CCK?I<<K<JKJN@CC<HL8C

/FI<:FM<IK?<GIF989@C@KP=IFDK?<F;;JN<ALJKNI@K<

/?@J8CN8PJNFIBJI<>8I;C<JJF=?FNK?<F;;JI8K@FJ8I<NI@KK<E@<@JALJKK?<J8D<8JFI $K;F<JEcKD8KK<I@EN?8KFI;<IK?<K<JKJ8I<8;D@E@JK<I<;FI@EN?8KFI;<IK?<I<JLCKJ8I<:FDGLK<; /?<GIFF=@JC<=K8J8E<O<I:@J<=FIK?<I<8;<I

/%8PE<J@E`+IF989@C@KP/?<FIP2@K?GGC@:8K@FEJ@E.:@<E:<8E;E>@E<<I@E>aJL>><JKJK?8K:I<;@9@C@KP8E;<M@;<E:<J?FLC;9<D<8JLI<;@E;<:@9<CJ

<:@9<CJ

<:@9<CJ8I<LJ<;=FID<8JLI@E><OGFE<EK@8C;@==<I<E:<JF= @EK<EJ@KP !FI<O8DGC<@=K?<JFLE;=IFD8E8LKFDF9@C<?FIE:8II@<JK@D<J8JDL:?<E<I>PhG<IJHL8I<D<K<IG<IJ<:FE;i8JK?<JFLE;=IFD8E8C8ID:CF:BK?<8LKFDF9@C<?FIENFLC;9<;<:@9<CJCFL;<I /?<JFLE;F=89@I;J@E>@E>D@>?K:8IIPK@D<JC<JJ<E<I>PK?8E8E8C8ID:CF:B8E;?<E:<NFLC;9<;<:@9<CJJF=K<I /F><KK?<ELD9<IF=;<:@9<CJPFLK8B<K?<CF>8I@K?D98J<8E;DLCK@GCP9P

9:8>7:@G@C< >BH:BG>HMor 

>BH:BG>HM9:8>7:@G

.LGGFJ<N<JK8IKN@K?8GI@FIGIF989@C@KPF=nK?8K8NFD8E?8J9I<8JK:8E:<I:FII<JGFE;@E>KF8EF;;JI8K@FF= E;K?<EN<8;D@E@JK<IK?I<<K<JKJF=C@B<C@?FF;I8K@FJ8E; 4FLcould DLCK@GCPK?FJ<ELD9<IJ_FIPFL:FLC;ALJK8;;K?<@ICF>8I@K?DJ

 &! &,-!,!.0($&,!'&'1+,'*%

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 64/309

@C<

@C<

@C< @C<

$KJK8IKJFLK8J=8@ICPLEC@B<CPK?8K8NFD8E?8J9I<8JK:8E:<IeFLI:I<;@9@C@KPC<M<C@J8Ke;<:@9<CJ /?<EK?I<<K<JKI<JLCKJ:FD<@E:FII<JGFE;@E>KF8E;;<:@9<CJF=<M@;<E:< /?@JI8@J<JK?<:I<;@9@C@KPC<M<C9P8KFK8CF=;<:@9<CJD<8E@E>K?8KK?<GI@FI:I<;@9@C@KPF=e;<:@9<CJ>F<JKF8GFJK<I@FI:I<;@9@C@KPF=;<:@9<CJ .FK?<F;;J>F=IFDKF8E;K?<GIF989@C@KP>F<J

=IFDnKF8IFLE;n

$E=IFEKF=PFL@J89FFB98>:FEK8@E@E>GFB<I:?@GJ $JK8IK<;FLKN@K?KNFJL:?9FFB98>JFE<:FEK8@E@E>I<;8E;9CL<:?@GJK?<FK?<I:FEK8@E@E>I<;8E;9CL< $=C@GG<;8=8@I:F@EKF;<K<ID@E<N?@:?9FFB98>KFLJ<JFPFLIGI@FIGIF989@C@KPK?8KK?<9FFB98>@E=IFEKF= 

 PFL@JK?<I<;9FFB98>@Jn )FNPFLJ8DGC<I8E;FDCP N@K?I<GC8:<D<EK8=K<I<8:?:?@G $EJ8DGC<JPFL><KI<;J8E;9CL<J 2?8K@JK?<GIF989@C@KPK?8KK?@J@JK?<GI<;FD@E8EKCPI<;98>

 %LJK=FI=LEKIP8E;NFIBK?@JFE<FLK@EPFLI?<8; 4FL;FEcKE<<;KF9<<O8:Ke8IFL>?<JK@D8K<@J>FF;<EFL>? 2?<EPFLcI<I<8;P:FEK@EL<FEN8I;

 ::FI;@E>KF8JKL;PG<I=FID<;9P'8NI<E:<+?@CC@GJ8E;28I;;N8I;J@EDFJKG<FGC<=8:<;N@K?K?@JGIF9C<D>@M<8E8EJN<I@EK?<I8E><nKFn @;PFL>@M<8JL9JK8EK@8CCP ?@>?<IGIF989@C@KPK?8EK?8K $=PFL;@;:FE>I8KLC8K@FEJe28I;;N8I;JNIFK<K?8KM<IPJ<C;FD;F<J8G<IJFE8EJN<IK?@JHL<JK@FEGIFG<ICP<M<E@=K?<G<IJFE@JI<C8K@M<CP=8D@C@8IN@K?8P<J@8EI<8JFE@E> /?<:FII<:K8EJN<I@Jn

%(&,**!,'*1

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 65/309

/?<C@B<C@?FF;I8K@F=FIK?<K<JKI<JLCK I<;:?@Ga@JN?@C<K?<C@B<C@?FF;I8K@F=FIK?<K<JKI<JLCK`9CL<:?@Ga@J /?<I<=FI<8 9CL<:?@G@J<O8:KCPK?<J8D<8DFLEKF=<M@;<E:<8J8I<;:?@GALJK

@EK?<FK?<I;@I<:K@FEe8I<;:?@G@J;<:@9<CJF=<M@;<E:<=FIK?<I<;98>8E;89CL<:?@G@Je;<:@9<CJF=<M@;<E:< $=PFL;I8N FE<9CL<:?@G8E;FE<I<;:?@GK?<P:8E:<CFLK .FK?< ratio F=I<;:?@GJKF9CL<:?@GJ;F<JEFKD8KK<IFECPK?< excess F=I<;:?@GJFM<I9CL<:?@GJD8KK<IJ /?<I<N<I<<@>?KI<;:?@GJ8E;=FLI9CL<:?@GJ@EKN<CM<J8DGC<JK?<I<=FI<=FLI more I<;:?@GJK?8E9CL<:?@GJ /?LJK?<GFJK<I@FIF;;JN@CC9<

 N?@:?@J8IFLE;@<8IFLE;n

/?<GI@FI:I<;@9@C@KPJK8IKJ8K;<:@9<CJ8E;K?<I<cJ8KFK8CF= 8IFLE;;<:@9<CJF=<M@;<E:<8E;@E;<<;K?@J:FII<JGFE;JKFF;;JF=8IFLE;FI8IFLE;n >8@EK?<I<cJJFD<IFLE;@E> <IIFI9LK@=PFLG<I=FID<;K?<FG<I8K@FEJLJ@E><O8:K8I@K?D<K@:K?<I<JLCKJNFLC;9<@;<EK@:8C

2<:8EEFNJ<< intuitively K?8KK?<9FFB98>GIF9C<DNFLC;?8M<<O8:KCPK?<J8D<8EJN<IF9K8@E<;@EALJKK?<J8D<N8P@=J@OK<<E:?@GJN<I<J8DGC<;8E;N<=FLE;K<EI<;:?@GJ8E;J@O9CL<:?@GJ

4FL8I<8D<:?8E@:=FI>@QDFJ 2?<E8>@QDFJKFGJ NFIB@E>@K@J;L<KF89CF:B<;?FJ<nF=K?<K@D< $=8  >@QDFcJ?FJ<@J9CF:B<;K?<I<@J8nGIF989@C@KPK?8KGIF;;@E>K?<>@QDFN@CCGIF;L:<JG8IBJ $=8>@QDFcJ?FJ<@J

LE9CF:B<;K?<I<@JFECP8n:?8E:<K?8KGIF;;@E>K?< >@QDFN@CCGIF;L:<JG8IBJ :LJKFD<I9I@E>JPFL8 D8C=LE:K@FE@E>>@QDF 4FLGIF;K?<>@QDF8E;=@E;K?8K@KGIF;L:<JJG8IBJ 2?8K@JK?<GIF989@C@KPK?8K8JG8IBeGIF;L:@E>>@QDF?8J89CF:B<;?FJ<

2?8K@JK?<J<HL<E:<F=8I@K?D<K@:8CFG<I8K@FEJK?8KPFLG<I=FID<;KFJFCM<K?@JGIF9C<D

hnnihnnnni

 &! &,-!,!.0($&,!'&'1+,'*%

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 66/309

.@D@C8ICPKF=@E;K?<:?8E:<K?8K8NFD8EN@K?GFJ@K@M<D8DDF>I8G?P?8J9I<8JK:8E:<IN<:FDGLK<;

DDCG>H>J:P86B8:FD86B8:F

55555555555555555555555555555555555555555555555

DDCG>H>J:P86B8:FD86B8:F

DDCG>H>J:PR86B8:FDR86B8:F

which isDDCG>H>J:86B8:F2DDCG>H>J:86B8:F

DDCG>H>J:R86B8:F3

which isDDCG>H>J:86B8:FDDCG>H>J:

which isD86B8:FPDCG>H>J:

/?<=LCCP><E<I8C=FIDF=K?@J:8C:LC8K@FE@JBEFNE8JBayes’ Theorem FI Bayes’ Rule:

8P<J/?<FI<D

DP0D0PD

D0PDD0PRDR

"@M<EJFD<G?<EFD<EFEK?8KN<N8EKKF@EM<JK@>8K<8E;8EF9J<IM8K@FE3K?8K@J<M@;<E:<89FLKe=FI<O8DGC<@EK?<GI<M@FLJ<O8DGC<@J9I<8JK:8E:<I8E;3@J8GFJ@K@M<D8DDF>I8G?Pe8P<Jc/?<FI<DK<CCJLJ?FNN<J?FLC; update FLIGIF989@C@KPF=>@M<EK?< new evidence 3

%(&,**!,'*1

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 67/309

PK?@JGF@EK8P<Jc/?<FI<DD8PJ<<D9C8K8EKCPF9M@FLJFI<M<EK8LKFCF>@:8CI8K?<IK?8E<O:@K@E>8E;E<N $=JFK?@J@EKIF;L:K@FE

?8J entirely succeeded @E@KJGLIGFJ<

Fun Fact!

Q. Who originally discovered Bayes’Theorem?A. /?<-<M<I<E;/?FD8J8P<J9P=8IK?<DFJK<E@>D8K@:=@>LI<@ED8K?<D8K@:8C?@JKFIP CDFJKEFK?@E>@JBEFNEF= 

8P<JcJC@=<8E;M<IP=<NF=?@JD8ELJ:I@GKJJLIM@M<; /?FD8J8P<JN8J9FIE@EFIKF%FJ?L88P<J8E;EE8IG<EK<I8E;?@J;8K<F=;<8K?@JC@JK<;8J /?<<O8:K;8K<F=/?FD8J8P<JcJ9@IK?@JEFKBEFNE=FI:<IK8@E9<:8LJ<%FJ?L88P<JK?FL>?8JLIGI@J@E>CPN<8CK?PD8EN8J8 D<D9<IF=8ELELJL8C<JFK<I@:8E;<M<E?<I<K@:8CI<C@>@FLJJ<:KK?<

`)FE:FE=FID@JKJa /?<)FE:FE=FID@JKJB<GKK?<@I9@IK?I<>@JK<IJJ<:I<KJLGGFJ<;CP =IFD=<8IF=I<C@>@FLJ;@J:I@D@E8K@FE

 N?8K<M<IK?<I<8JFEEFKIL<I<:FI;<O@JKJF=/?FD8J8P<JcJ9@IK? /?FD8J8P<J

 N8JI8@J<;8)FE:FE=FID@JK8E;N8JJFFEGIFDFK<;@EKFK?<?@>?<II8EBJF=K?<)FE:FE=FID@JKK?<FJFG?<IJN?<E:<:FD<JK?<`-<M<I<E;a@E?@JE8D<

$E8P<JN8J<C<:K<;8!<CCFNF=K?<-FP8C.F:@<KPF='FE;FEK?<DFJKGI<JK@>@FLJJ:@<EK@=@:9F;PF=@KJ;8P;<JG@K<8P<J?8M@E>GL9C@J?<;EFJ:@<EK@=@:FID8K?<D8K@:8CNFIBJ8KK?8KK@D<8P<JcJEFD@E8K@FE:<IK@=@:8K<N8JJ@>E<;9PJGFEJFIJ@E:CL;@E>K?<+I<J@;<EK8E;K?<.<:I<K8IPF=K?<.F:@<KPD8B@E>?@J

 &! &,-!,!.0($&,!'&'1+,'*%

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 68/309

<C<:K@FE8CDFJK:<IK8@E M<EKF;8P?FN<M<I@KI<D8@EJ8DPJK<IP why JL:?

 N<@>?KPE8D<JJGFEJFI<;8ELEBEFNE@EKF

K?<-FP8C.F:@<KP

8P<JcJJFC<GL9C@:8K@FE;LI@E>?@JBEFNEC@=<K@D<N8J8CC<><;CP8DPJK@:8C9FFB<EK@KC<; Divine Benevolence, C8P@E>=FIK?K?<FI@>@E8C:8LJ8K@FE8E;LCK@D8K<GLIGFJ<F= K?<LE@M<IJ< /?<9FFB@J:FDDFECP 8KKI@9LK<;KF8P<JK?FL>?@K@JJ8@;K?8KEF8LK?FI8GG<8I<;FEK?<K@KC<G8><8E;K?<<EK@I<NFIB@JJFD<K@D<J:FEJ@;<I<;KF9<F=;L9@FLJGIFM<E8E:<

(FJKDPJK<I@FLJF=8CC8P<Jc/?<FI<D@KJ<C=8GG<8IJ@E88P<JD8ELJ:I@GKGI<J<EK<;KFK?<-FP8C.F:@<KPF='FE;FE@E three years after Bayes’s supposed deathin 1761! 

<JG@K<K?<J?F:B@E>:@I:LDJK8E:<JF=@KJGI<J<EK8K@FE8P<Jc/?<FI<DN8JJFFE=FI>FKK<E8E;N8JGFGLC8I@Q<;N@K?@EK?<J:@<EK@=@::FDDLE@KPFECP9PK?<C8K<I<==FIKJF=K?<>I<8KD8K?<D8K@:@8E+@<II<e.@DFE'8GC8:< '8GC8:<?@DJ<C=@J8CDFJK8J<E@>D8K@:8J8P<JN<;FEcK<M<EBEFN 

 N?<K?<I@KN8J`+@<II<aFI`.@DFEaK?8KN8J

?@J8:KL8C=@IJKE8D< '8GC8:<cJG8G<IJ8I<J8@;KF?8M<:FEK8@E<;8;<J@>E=FI8E$:8G89C<F=GI<;@:K@E>8CC=LKLI<<M<EKJK?<JFe:8CC<;`'8GC8:@8EJLG<I@EK<CC@><E:<a2?@C<@K@J><E<I8CCP9<C@<M<;K?8K'8GC8:<E<M<IKI@<;KF@DGC<D<EK?@J;<J@>EK?<I<I<D8@EJK?<=8:KK?8K'8GC8:<GI<J:@<EKCP =C<;K?<>L@CCFK@E<K?8K:C8@D<;D8EPF=?@J:FCC<8>L<J;LI@E>K?<-<@>EF=/<IIFI

%(&,**!,'*1

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 69/309

M<EKF;8PG?PJ@:@JKJJFD<K@D<J8KKI@9LK<LELJL8C<==<:KJKF8`'8GC8:@8E*G<I8KFIa@EK<IM<E@E>@EK?<@I<OG<I@D<EKJ

$EJLDD8IPN<;FEFKBEFNK?<I<8C:@I:LDJK8E:<JF=8P<JcJ9@IK?K?<LCK@D8K<FI@>@EJF=8P<Jc/?<FI<D8P<JcJ8:KL8C

 P<8IF=;<8K?FI<M<EN?<K?<I8P<J<M<II<8CCP;@<; )FE<K?<C<JJ`-<M<I<E;/?FD8J8P<JaN?8K<M<I?@JKIL<@;<EK@KP?8JK?<>I<8K<JK=FE;E<JJ8E;>I8K@KL;<F= 8IK?cJJ:@<EK@=@::FDDLE@KP

.FN?P@J@KK?8KJFD<G<FGC<8I<JF excited 89FLK8P<Jc/?<FI<D

`FPFL9<C@<M<K?8K8EL:C<8IN8IN@CCF::LI@EK?<E<OKP<8IJ$=EFN?PEFKa .@E:<$N8EK<;KFLJ<JFD<:FDDFE8EJN<IJKF

K?@JHL<JK@FEKFD8B<8GF@EK89FLKI8K@FE8C@KP$N<EK8?<8;8E;8JB<;K?<89FM<HL<JK@FE@E8E$-:?8EE<CG?@CFJFG?PFE

!)<K

*E<!)<KK<IN?F8EJN<I<;I<GC@<;`)FaKFK?<89FM<HL<JK@FE9LK8;;<;K?8K?<9<C@<M<;9@FCF>@:8CN8I=8I<NFLC;N@G<FLK`naF=?LD8E@KPN@K?@EK?<E<OKK<EP<8IJ $K?<E8JB<;

 N?<K?<I?<9<C@<M<;nN8J8GFJJ@9@C@KP a)Fa?<J8@; a2?P EFKa$8JB<; a<:8LJ<$cD8EFGK@D@JKa?<J8@; h-F8EFB<F= G?@CFJFG?PFE!)<KN@J?<JKF9<:I<;@K<;N@K?K?@JJK8K<D<EK<M<E?8M@E>9<<EN8IE<;K?8K@KN@CCEFK9<:8JK@E8 :FDGC@D<EK8IPC@>?K "FF;=FI?@Di EFK?<IG<IJFEN?F8EJN<I<;K?<89FM<HL<JK@FEJ8@;K?8K?<;@;EcK<OG<:K8EL:C<8I

 N8I=FIP<8IJ9<:8LJ<`CCF=K?<GC8P<IJ@EMFCM<;@E;<:@J@FEJI<>8I;@E>EL:C<8IN8I8I<EFK@EK<I<JK<;I@>?KEFNa aLKN?P <OK<E;K?8KFLK=FIP<8IJa$8JB<; a+LI<?FG<aN8J?@JI<GCP

2?8K@J@K exactly K?8KD8B<JK?<J<K?FL>?KJ`@II8K@FE8Cae8GFFI

 N8PF=8II@M@E>8KKILK? /?<I<8I<8ELD9<IF=@EKL@K@M<I<GC@<J

 &! &,-!,!.0($&,!'&'1+,'*%

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 70/309

K?8K:8E9<>@M<EKFK?@J=FI<O8DGC< `$K@JEFKI8K@FE8CKF9<C@<M<K?@E>JFECP9<:8LJ<K?<P8I<:FD=FIK@E>a *=:FLIJ<@K@J<HL8CCP @II8K@FE8CKF9<C@<M<K?@E>JFECP9<:8LJ<K?<P8I< discomforting; K?<

J<:FE;<IIFI@JC<JJ:FDDFE9LK<HL8CCP@II8K@FE8C *K?<I@EKL@K@M<8I>LD<EKJ@E:CL;<K?<@;<8K?8K`2?<K?<IFIEFKPFL?8GG<EKF9<8EFGK@D@JK?8JEFK?@E>KF;FN@K?N?<K?<I9@FCF>@:8CN8I=8I<N@G<JFLKK?<?LD8EJG<:@<JaFI`+LI<?FG<@JEFK<M@;<E:<89FLKEL:C<8IN8I9<:8LJ<@K@JEFK8EF9J<IM8K@FE89FLKEL:C<8IN8Ia

/?<I<@J8CJF8D8K?<D8K@:8CI<GCPK?8K@JGI<:@J<<O8:K8E;:FEK8@EJ8CCK?<@EKL@K@FEJ8JJG<:@8C:8J<J /?@JD8K?<D8K@:8C

I<GCP@JBEFNE8J8P<Jc/?<FI<D

!FI<O8DGC<K?<I<GCP`2?<K?<IFIEFKPFL?8GG<EKF9<8EFGK@D@JK?8JEFK?@E>KF;FN@K?N?<K?<I9@FCF>@:8CN8I=8I<N@G<JFLKK?<?LD8EJG<:@<Ja:8E9<KI8EJC8K<;@EKFK?<JK8K<D<EK

GhPFL8I<:LII<EKCP8EFGK@D@JKR9@FCF>@:8CN8IF::LIJN@K?@EK<E P<8IJ8E;N@G<JFLK?LD8E@KPiGhPFL8I<:LII<EKCP8EFGK@D@JKR9@FCF>@:8CN8IF::LIJN@K?@EK<E

 P<8IJ8E;;F<JEFKN@G<FLK?LD8E@KPi

.@E:<K?<KNFGIF989@C@K@<J=FI D0P 8E; D0PR 8I<<HL8C8P<Jc/?<FI<DJ8PJK?8K DP0D8JN<?8M<<8IC@<IJ<<EN?<EK?<KNF:FE;@K@FE8CGIF989@C@K@<J8I<<HL8CK?<I<M@J<;GIF989@C@KP<HL8CJK?<GI@FIGIF989@C@KP $=38E;8I<LE:FEE<:K<;eJK8K@JK@:8CCP@E;<G<E;<EKeK?<E=@E;@E>K?8K3@JKIL<:8EEFK9<<M@;<E:<K?8K@JKIL<F9J<IM@E>3;F<JEFKLG;8K<FLIGIF989@C@KP=FIJ8P@E>`3a@JEFK8E8I>LD<EK=FI

LKJLGGFJ<PFL8I<8I>L@E>N@K?JFD<FE<N?F@JM<I98CCP:C<M<I8E;N?FJ8PJJFD<K?@E>C@B<`?9LKJ@E:<$cD8EFGK@D@JK$cCC?8M<I<E<N<;?FG<=FIKFDFIIFNNFIB8C@KKC<?8I;<I8KDP;<8;e<E;AF9GLDGLGK?<>CF98C<:FEFDP8C@KKC<<M<EKL8CCPK?IFL>?K?<KI@:BC<e;FNE<==<:KJ<E;@E>8=<N;FCC8IJ@EKFK?<GF:B<KF= K?<I<J<8I:?<IN?FLCK@D8K<CP=@E;J8N8PKFJKFG9@FCF>@:8C

 N8I=8I<eJFPFLJ<<K?<KNF<M<EKJ8I<I<C8K<;8=K<I8CC8E;$:8E

LJ<FE<8JM8C@;<M@;<E:<89FLKK?<FK?<Ia $EFE<J<EJ<K?@J@J

%(&,**!,'*1

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 71/309

:FII<:Ke any :FII<C8K@FEEFD8KK<I?FNN<8B@J=8@IGI<P=FI8P<Jc/?<FI<D but 8P<Jc/?<FI<D;@JK@E>L@J?<J9<KN<<EN<8B8E;JKIFE><M@;<E:< /?8K@J8P<Jc/?<FI<DEFKFECPK<CCJLJ

 N?8K@J8E;@JEcK<M@;<E:<@K8CJF;<J:I@9<JK?< strength F= <M@;<E:< 8P<Jc/?<FI<DEFKFECPK<CCJLJ when KFI<M@J<FLIGIF989@C@K@<J9LK how much KFI<M@J<FLIGIF989@C@K@<J :FII<C8K@FE9<KN<<E?FG<8E;9@FCF>@:8CN8I=8I<D8P<O@JK9LK@KcJ8CFKN<8B<IK?8EK?<JG<8B<IN8EKJ@KKF9<?<@JI<M@J@E>?@JGIF989@C@K@<JDL:?KFF=8I

'<KcJJ8PPFLcI<8NFD8EN?FcJALJKLE;<I>FE<8D8DDF>I8G?P+I<M@FLJCPPFL=@>LI<;K?8KPFL?8;8M<IPJD8CC:?8E:<F=?8M@E> 

9I<8JK:8E:<IN<cCCJLGGFJ<K?8KPFLI<8;K?<JK8K@JK@:JJFD<N?<I<8E;JFPFLBEFNK?<:?8E:<@Jn 2?<EK?<GFJ@K@M<D8DDF>I8G?P:FD<J@EPFLI<JK@D8K<;:?8E:<J?FLC;EFNJ?@=KKFn /?<I<@JEFIFFDKFJ8PJFD<K?@E>C@B<`*?N<CC8 GFJ@K@M<D8DDF>I8G?P@JEcK;<=@E@K<<M@;<E:<JFD<?<8CK?P 

 NFD<E><KGFJ@K@M<D8DDF>I8G?@<JKFF $;FEcKN8EKKF;<JG8@IKFF<8ICP8E;$cDEFK>F@E>KFI<M@J<DPGIF989@C@KPLEK@CDFI<<M@;<E:<:FD<J@E 2?P <:8LJ<$cD8FGK@D@JKa E;K?<I<@JJ@D@C8ICPEFIFFD=FIJ8P@E>`2<CC8GFJ@K@M<D8DDF>I8G?PD8P 

EFK9<;<=@E@K<<M@;<E:<9LK$cD>F@E>KF8JJLD<K?<NFIJKLEK@C$=@E;FK?<IN@J< 2?P <:8LJ<$cD8G<JJ@D@JKa 4FLII<M@J<;GIF989@C@KPJ?FLC;>FKF nEFDFI<EFC<JJ

8P<Jc/?<FI<D;<J:I@9<JN?8KD8B<JJFD<K?@E>`<M@;<E:<a8E;?FNDL:?<M@;<E:<@K@J .K8K@JK@:8CDF;<CJ8I<AL;><;9P :FDG8I@JFEKFK?< Bayesian method 9<:8LJ<@EJK8K@JK@:JK?<8P<J@8ED<K?F;@J8J>FF;8J@K><KJeK?<8P<J@8ED<K?F;;<=@E<JK?<D8O@DLD8DFLEKF=D@C<8><PFL:8E><KFLKF=8>@M<EG@<:<F=<M@;<E:<@EK?<J8D<N8PK?8KK?<IDF;PE8D@:J;<=@E<JK?<D8O@DLD8DFLEKF=NFIBPFL:8E><KFLKF=8K<DG<I8KLI<;@==<I<EK@8C /?@J@JN?PPFL?<8I:F>E@K@M<J:@<EK@JKJK8CB@E>89FLK Bayesian reasoners $E:F>E@K@M<J:@<E:< Bayesian reasoner @JK?<K<:?E@:8CCPGI<:@J<:F;<NFI;K?8KN<LJ<KFD<8E rational mind.

/?<I<8I<8CJF8ELD9<IF=><E<I8C?<LI@JK@:J89FLK?LD8EI<8JFE@E>K?8KPFL:8EC<8IE=IFDCFFB@E>8K8P<Jc/?<FI<D

 &! &,-!,!.0($&,!'&'1+,'*%

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 72/309

!FI<O8DGC<@ED8EP;@J:LJJ@FEJF=8P<Jc/?<FI<DPFLD8P?<8I:F>E@K@M<GJP:?FCF>@JKJJ8P@E>K?8KG<FGC< do not take prior  frequencies sufficiently into account, D<8E@E>K?8KN?<EG<FGC<

8GGIF8:?8GIF9C<DN?<I<K?<I<cJJFD<<M@;<E:<3@E;@:8K@E>K?8K:FE;@K@FED@>?K?FC;KIL<K?<PK<E;KFAL;><cJC@B<C@?FF;JFC<CP9P?FNN<CCK?<<M@;<E:<3J<<DJKFD8K:?N@K?FLKK8B@E>@EKF8::FLEKK?<GI@FI=I<HL<E:PF= $=PFLK?@EB=FI<O8DGC<K?8KLE;<IK?<D8DDF>I8G?P<O8DGC<K?<NFD8EcJ:?8E:<F=?8M@E>9I<8JK:8E:<I@J@EK?<I8E><F=nenK?<EK?@JB@E;F=I<8JFE@E>@J@EJ<EJ@K@M<KFK?<GI@FI=I<HL<E:P>@M<E@EK?<GIF9C<D@K;F<JEcKEFK@:<N?<K?<InF=NFD<EFInF= 

 NFD<EJK8IKFLK?8M@E>9I<8JK:8E:<I `+8PDFI<8KK<EK@FEKFK?<

GI@FI=I<HL<E:Pa@JFE<F=K?<D8EPK?@E>JK?8K?LD8EJE<<;KF9<8I@ED@E;KFG8IK@8CCP:FDG<EJ8K<=FIFLI9L@CKe@E@E8;<HL8:@<J

 I<C8K<;<IIFI@JKFG8PKFFDL:?8KK<EK@FEKFGh3Ri8E;EFK<EFL>?KFGh3RSiN?<E;<K<ID@E@E>?FNDL:?<M@;<E:<3@J=FI

  /?<;<>I<<KFN?@:?8I<JLCK3@J evidence for A ;<G<E;JEFKFECPFEK?<JKI<E>K?F=K?<JK8K<D<EK we’d expect to see result X if Awere true, 9LK8CJFFEK?<JKI<E>K?F=K?<JK8K<D<EK we wouldn’t expect to see result X if A weren’t true. !FI<O8DGC<@=@K@JI8@E@E>

K?@JM<IPJKIFE>CP@DGC@<JK?<>I8JJ@JN<K DK:H<F6GGPF6>BR e9LKJ<<@E>K?8KK?<>I8JJ@JN<K;F<JEcKE<:<JJ8I@CPD<8EK?8K@K?8JALJKI8@E<;G<I?8GJK?<JGI@EBC<IN8JKLIE<;FEFI

 PFLcI<CFFB@E>8KK?<<8ICPDFIE@E>;<N .@E:<DK:H<F6GGPRF6>B @JJL9JK8EK@8CCP>I<8K<IK?8EQ<IFDF6>BPK:H<F6GG @JJL9JK8EK@8CCPC<JJK?8EFE< *EK?<FK?<I?8E;@=K?<>I8JJN8J never N<KN?<E@KN8JEcKI8@E@E>K?<EBEFN@E>K?8KK?<>I8JJN8JN<KNFLC; always J?FNK?8K@KN8J

I8@E@E> DF6>BPK:H<F6GGR<M<E@= DK:H<F6GGPF6>BK?8K@J<M<E@=K?<>I8JJFECP>FK N<KnF=K?<K@D<J@KI8@E<; M@;<E:<@J8CN8PJK?<I<JLCKF=K?< differential 9<KN<<EK?<KNF:FE;@K@FE8CGIF989@C@K@<J Strong <M@;<E:<@JEFKK?<GIF;L:KF=8M<IP?@>?GIF989@C@KPK?8KC<8;JKF39LKK?<GIF;L:KF=8M<IP low GIF989@C@KPK?8K not-A :FLC;?8M<C<;KF3

/?< Bayesian revolution in the sciences @J=L<C<;EFKFECP9PDFI<8E;

DFI<:F>E@K@M<J:@<EK@JKJJL;;<ECPEFK@:@E>K?8KD<EK8C

%(&,**!,'*1

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 73/309

G?<EFD<E8?8M<8P<J@8EJKIL:KLI<@EK?<DEFKFECP9PJ:@<EK@JKJ@E<M<IP=@<C;C<8IE@E>KFAL;><K?<@IJK8K@JK@:8CD<K?F;J9P :FDG8I@JFEN@K?K?<8P<J@8ED<K?F;9LK8CJF9PK?<@;<8K?8K

 science itself is a special case of Bayes’ Theorem; experimental evidence is Bayesian evidence. /?<8P<J@8EI<MFCLK@FE8I@<J?FC;K?8KN?<EPFLG<I=FID8E<OG<I@D<EK8E;><K<M@;<E:<K?8K`:FE=@IDJaFI`;@J:FE=@IDJaPFLIK?<FIPK?@J:FE=@ID8K@FE8E;;@J:FE=@ID8K@FE@J>FM<IE<;9PK?<8P<J@8EILC<J !FI<O8DGC<PFL?8M<KFK8B<@EKF8::FLEKEFKFECPN?<K?<IPFLIK?<FIPGI<;@:KJK?<G?<EFD<EFE9LKN?<K?<IFK?<IGFJJ@9C<<OGC8E8K@FEJ8CJFGI<;@:KK?<G?<EFD<EFE +I<M@FLJCPK?<DFJKGFGLC8IG?@CFJFG?PF= J:@<E:<N8JGIF989CP&8IC+FGG<IcJ falsificationism eK?@J@JK?<FC;

G?@CFJFG?PK?8KK?<8P<J@8EI<MFCLK@FE@J:LII<EKCP;<K?IFE@E>&8IC+FGG<IcJ@;<8K?8KK?<FI@<J:8E9<;<=@E@K<CP=8CJ@=@<;9LKE<M<I;<=@E@K<CP:FE=@ID<;@JP<K8EFK?<IJG<:@8C:8J<F=K?<8P<J@8EILC<J@= D0PR e@=K?<K?<FIPD8B<J8;<=@E@K<GI<;@:K@FEeK?<EF9J<IM@E>S3M<IPJKIFE>CP=8CJ@=@<J *EK?<FK?<I?8E;@= D0PR 8E;N<F9J<IM<3K?@J;F<JEcK;<=@E@K<CP:FE=@IDK?<K?<FIPK?<I<D@>?K9<JFD<FK?<I:FE;@K@FEJL:?K?8K D0PR@EN?@:?:8J<F9J<IM@E>3;F<JEcK=8MFIFM<I !FIF9J<IM@E>3KF;<=@E@K<CP:FE=@ID

 N<NFLC;?8M<KFBEFNEFKK?8K D0PR9LKK?8KD0PRRN?@:?@JJFD<K?@E>K?8KN<:8EcKBEFN9<:8LJ<

 N<:8EcKI8E><FM<I8CCGFJJ@9C<8CK<IE8K@M<<OGC8E8K@FEJ !FI<O8DGC<N?<E@EJK<@EcJK?<FIPF="<E<I8C-<C8K@M@KPKFGGC<;)<NKFEcJ@E:I<;@9CPN<CCe:FE=@ID<;K?<FIPF=>I8M@KP@KKLIE<;FLKK?8K8CCF=)<NKFEcJGI<;@:K@FEJN<I<ALJK8JG<:@8C:8J<F= @EJK<@EcJGI<;@:K@FEJ

4FL:8E<M<E=FID8C@Q<+FGG<IcJG?@CFJFG?PD8K?<D8K@:8CCP /?<C@B<C@?FF;I8K@F=FI3 D0PD0PR;<K<ID@E<J?FNDL:?F9J<IM@E>3JC@;<JK?<GIF989@C@KP=FIK?<C@B<C@?FF;I8K@F@J

 N?8KJ8PJ how strong 3@J8J<M@;<E:< 2<CC@EPFLIK?<FIPPFL:8EGI<;@:K3N@K?GIF989@C@KP@=PFLC@B<9LKPFL:8EcK:FEKIFCK?<;<EFD@E8KFIF=K?<C@B<C@?FF;I8K@F D0PR eK?<I<N@CC8CN8PJ9<JFD<8CK<IE8K@M<K?<FI@<JK?8K8CJFGI<;@:K38E;N?@C<

 N<>FN@K?K?<J@DGC<JKK?<FIPK?8K=@KJK?<:LII<EK<M@;<E:<PFLD8PJFD<;8P<E:FLEK<IJFD<<M@;<E:<K?8K8E8CK<IE8K@M<K?<FIP 

GI<;@:KJ9LKPFLIK?<FIP;F<JEFK /?8KcJK?<?@;;<E>FK:?8K?8K

 &! &,-!,!.0($&,!'&'1+,'*%

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 74/309

KFGGC<;)<NKFEcJK?<FIPF=>I8M@KP .FK?<I<cJ8C@D@KFE?FN DL:?D@C<8><PFL:8E><K=IFDJL::<JJ=LCGI<;@:K@FEJK?<I<cJ8 C@D@KFE?FN?@>?K?<C@B<C@?FF;I8K@F>F<J=FI confirmatory

<M@;<E:<

*EK?<FK?<I?8E;@=PFL<E:FLEK<IJFD<G@<:<F=<M@;<E:<4K?8K@J;<=@E@K<CP not GI<;@:K<;9PPFLIK?<FIPK?@J@J enormouslyJKIFE><M@;<E:< against PFLIK?<FIP $= D1P @J@E=@E@K<J@D8CK?<EK?<C@B<C@?FF;I8K@FN@CC8CJF9<@E=@E@K<J@D8C !FI<O8DGC<@= D1P @Jn8E; D1PR @JnK?<EK?<C@B<C@?FF;I8K@F D1PD1PR N@CC9< e;<:@9<CJF= <M@;<E:< *I=C@GG@E>K?<C@B<C@?FF;I8K@F@= D1P @J very small,

K?<E D1PRD1P N@CC9< very large, D<8E@E>K?8KF9J<IM@E> 4>I<8KCP=8MFIJSFM<I !8CJ@=@:8K@FE@JDL:?JKIFE><IK?8E:FE=@ID8K@FE /?@J@J8:FEJ<HL<E:<F=K?<<8IC@<IGF@EKK?8K very strong <M@;<E:<@JEFKK?<GIF;L:KF=8M<IP?@>?GIF989@C@KPK?8KC<8;JKF39LKK?<GIF;L:KF=8M<IP low GIF989@C@KPK?8K not-A:FLC;?8M<C<;KF3 /?@J@JK?<GI<:@J<8P<J@8EILC<K?8KLE;<IC@<JK?<?<LI@JK@:M8CL<F=+FGG<IcJ=8CJ@=@:8K@FE@JD

.@D@C8ICP+FGG<IcJ;@:KLDK?8K8E@;<8DLJK9<=8CJ@=@89C<:8E9<@EK<IGI<K<;8J8D8E@=<JK8K@FEF=K?<8P<J@8E:FEJ<IM8K@FEeF=eGIF989@C@KPILC<@=8I<JLCK3@JGFJ@K@M<<M@;<E:<=FIK?<K?<FIPK?<EK?<I<JLCKS3NFLC;?8M<;@J:FE=@ID<;K?<K?<FIPKFJFD<<OK<EK $=PFLKIPKF@EK<IGI<K9FK?38E;S38J`:FE=@ID@E>aK?<K?<FIPK?<8P<J@8EILC<JJ8PK?@J@J@DGFJJ@9C< /F@E:I<8J<K?<GIF989@C@KPF=8K?<FIPPFL must <OGFJ<@KKFK<JKJK?8K:8EGFK<EK@8CCP;<:I<8J<@KJGIF989@C@KPK?@J@JEFKALJK8ILC<=FI;<K<:K@E>NFLC;e9<:?<8K<IJ@EK?<JF:@8CGIF:<JJF=J:@<E:<9LK8 

:FEJ<HL<E:<F=8P<J@8EGIF989@C@KPK?<FIP *EK?<FK?<I?8E;+FGG<IcJ@;<8K?8KK?<I<@J only =8CJ@=@:8K@FE8E; no such thing 8J:FE=@ID8K@FEKLIEJFLKKF9<@E:FII<:K 8P<Jc/?<FI<DJ?FNJK?8K=8CJ@=@:8K@FE@J very strong <M@;<E:<:FDG8I<;KF:FE=@ID8K@FE9LK=8CJ@=@:8K@FE@JJK@CCGIF989@C@JK@:@EE8KLI<@K@JEFK>FM<IE<;9P =LE;8D<EK8CCP;@==<I<EKILC<J=IFD:FE=@ID8K@FE8J+FGG<I8I>L<;

.FN<=@E;K?8KD8EPG?<EFD<E8@EK?<:F>E@K@M<J:@<E:<JGCLJ

K?<JK8K@JK@:8CD<K?F;JLJ<;9P J:@<EK@JKJGCLJK?<J:@<EK@=@:

%(&,**!,'*1

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 75/309

D<K?F;@KJ<C=8I<8CCKLIE@E>FLKKF9<JG<:@8C:8J<JF=8P<Jc/?<FI<D #<E:<K?<8P<J@8EI<MFCLK@FE

Fun Fact!

Q. Are there any limits to the powerof Bayes’ Theorem?A. ::FI;@E>KFC<><E;FE<N?F=LCCP 

 >I8JG<;8P<Jc/?<FI<DNFLC;>8@EK?<89@C@KPKF:I<8K<8E;G?PJ@:8CCP<EK<I8E8CK<IE8K<LE@M<IJ<LJ@E>FECPF==eK?<eJ?<C= <HL@GD<EK8E;8J?FIK:FDGLK<IGIF>I8D

*E<N?F=LCCP>I8JGJ8P<Jc/?<FI<DP<KI<D8@EJ@EFLILE@M<IJ<KF8@;FK?<IJ@JBEFNE8J88P<J8KKM8

8P<J/?<FI<D

DP0D0PD

D0PDD0PRDR

2?PN8@KJFCFE>KF@EKIF;L:<8P<Jc/?<FI<D@EJK<8;F=ALJKJ?FN@E>@K8KK?<9<>@EE@E> 2<CC_9<:8LJ<$cM<KI@<;K?8K9<=FI<8E;N?8K?8GG<EJ@EDP<OG<I@<E:<@JK?8KG<FGC<><K8CCK8E>C<;LG@EKIP@E>KF8GGCP8P<Jc/?<FI<D8J8J<KF= poorly grounded 

 mental rules; @EJK<8;F=K?</?<FI<D?<CG@E>@K9<:FD<J one morething to juggle mentally, JFK?8K@E8;;@K@FEKFKIP@E>KFI<D<D9<I?FND8EPNFD<EN@K?9I<8JK:8E:<I?8M<GFJ@K@M<D8DDF>I8G?@<JK?<I<8;<I@J8CJFKIP@E>KFI<D<D9<IN?<K?<I@KcJD0P @EK?<ELD<I8KFIFI DP08E;N?<K?<I8GFJ@K@M<D8DDF>I8G?PI<JLCK:FII<JGFE;JKFFI38E;N?@:?J@;<F= D0P @JK?<@DGC@:8K@FE8E;N?8KK?<K<IDJ8I<@EK?<;<EFD@E8KFI8E;JFFE $EK?@J<O:IL:@8K@E>CP><EKC<@EKIF;L:K@FE$KI@<;KFJ?FN8CCK?<NFIB@E>JF=8P<J@8E

I<8JFE@E> without <M<I@EKIF;L:@E>K?<<OGC@:@K/?<FI<D8J

 &! &,-!,!.0($&,!'&'1+,'*%

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 76/309

JFD<K?@E><OKI8KFD<DFI@Q<?FG<=LCCPI<;L:@E>K?<ELD9<IF= =8:KFIJK?<I<8;<IE<<;<;KFD<EK8CCPAL>>C<

M<E@=PFL?8GG<EKF9<FE<F=K?<=FIKLE8K<G<FGC<N?F:8E<8J@CP>I8JG8E;8GGCP89JKI8:KK?<FI<DJK?<D<EK8CeAL>>C@E> GIF9C<D@JJK@CCJFD<K?@E>KF9<8I@ED@E;@=PFL<M<IE<<;KF<OGC8@E8P<J@8EI<8JFE@E>KFJFD<FE<<CJ<

$=PFL;F=@E;PFLIJ<C=CFJ@E>KI8:BDP8;M@:<@JKF=FI><K8P<Jc/?<FI<D8J8E equation 8E;K?@EB89FLKK?< graph. Ghi8E;GhSi8I<8KK?<KFG Gh3Ri8E;Gh3RSi8I<K?<GIFA<:K@FE=8:KFIJGh3i8E;Gh3Si8I<8KK?<9FKKFD E;GhR3i<HL8CJK?<

GIFGFIK@FEF=Gh3iN@K?@EGh3iGh3Si /?<>I8G?@JEcKJ?FNE?<I<e9LK:8EPFLJ<<@K@EPFLID@E;

 E;@=K?@EB@E>89FLKK?<>I8G?;F<JEcKNFIB$JL>><JK=FI><KK@E> 89FLK8P<Jc/?<FI<D<EK@I<CPeALJKKIPKFNFIBFLKK?<JG<:@=@:GIF9C<D@E>@QDFJ?FJ<J8E;JG8IBJFIN?8K<M<I@K@J

#8M@E>@EKIF;L:<;8P<Jc/?<FI<D<OGC@:@KCPN<:8E<OGC@:@KCP ;@J:LJJ@KJ:FDGFE<EKJ

DP0D0PD

D0PDD0PRDR

2<cCCJK8IKN@K?GhR3i $=PFL<M<I=@E;PFLIJ<C=><KK@E>:FE=LJ<;89FLKN?8KcJ8E;N?8KcJ3@E8P<Jc/?<FI<DJK8IKN@K?GhR3iFEK?<C<=KJ@;<F=K?<<HL8K@FEK?8KcJK?<J@DGC<JKG8IKKF

@EK<IGI<K @JK?<K?@E>N<N8EKKFBEFN89FLK 3@J?FNN<cI<F9J<IM@E>@K3@JK?<<M@;<E:<N<cI<LJ@E>KFD8B<@E=<I<E:<J89FLK -<D<D9<IK?8K=FI<M<IP<OGI<JJ@FEGh,R+iN<N8EKKFBEFN89FLKK?<GIF989@C@KP=FI,>@M<E+K?<;<>I<<KFN?@:?+@DGC@<J,e8DFI<J<EJ@9C<EFK8K@FEN?@:?@K@JEFNKFFC8K<KF8;FGKNFLC;9< D*)

Gh,R+i@J:CFJ<CPI<C8K<;KFGh,+i9LKK?<P8I<EFK@;<EK@:8COGI<JJ<;8J8GIF989@C@KPFI8=I8:K@FEGh,+i@JK?<GIFGFIK@FE

F=K?@E>JK?8K?8M<GIFG<IKP,8E;GIFG<IKP+N@K?@E all things; @<

%(&,**!,'*1

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 77/309

K?<GIFGFIK@FEF=`NFD<EN@K?9I<8JK:8E:<I8E;8GFJ@K@M<D8DDF>I8G?PaN@K?@EK?<>IFLGF= all women. $=K?<KFK8CELD9<IF=NFD<E@J8E;NFD<E?8M<9I<8JK:8E:<I8E;8 

GFJ@K@M<D8DDF>I8G?PK?<EGh,+i@Jn 4FLD@>?KJ8PK?8KK?<89JFCLK<HL8EK@KP@J9<@E>EFID8C@Q<;KF8 GIF989@C@KPI<C8K@M<KFK?< group of all women. *IKFD8B<@K:C<8I<IJLGGFJ<K?8KK?<I<cJ8>IFLGF=NFD<EN@K?9I<8JK:8E:<I8E;8GFJ@K@M<D8DDF>I8G?PN@K?@E8KFK8CJ8DGC<>IFLGF=

 NFD<E @JK?<89JFCLK<HL8EK@KP $=PFLG@:BFLK8I8E;FD NFD8E=IFDK?< entire sample, K?<EK?< probability PFLcCCG@:B8  NFD8EN@K?9I<8JK:8E:<I8E;8GFJ@K@M<D8DDF>I8G?P@JGh,+iFInh@EK?@J<O8DGC<i

*EK?<FK?<I?8E;Gh,R+i@JK?<GIFGFIK@FEF=K?@E>JK?8K?8M<GIFG<IKP,8E;GIFG<IKP+N@K?@E all things that have P; @<K?<GIFGFIK@FEF=NFD<EN@K?9I<8JK:8E:<I8E;8GFJ@K@M<D8DDF>I8G?PN@K?@EK?<>IFLGF= all women with positive mammographies. $=K?<I<8I<NFD<EN@K?9I<8JK:8E:<I8E;GFJ@K@M<D8DDF>I8G?@<JNFD<EN@K?GFJ@K@M<D8DDF>I8G?@<J8E;NFD<EK?<EGh,+i@JK?<GIF989@C@KPF=><KK@E>FE<F=K?FJ<NFD<E@=PFLcI<G@:B@E>8K

I8E;FD=IFDK?<<EK@I<>IFLGF=N?@C<Gh,R+i@JK?<GIF989@C@KPF=><KK@E>FE<F=K?FJ<NFD<E@=PFLcI<G@:B@E>8KI8E;FD=IFDK?<JD8CC<I>IFLGF=

$E8J<EJ< D*P)I<8CCPD<8EJ D*)P)9LKJG<:@=P@E>K?<<OKI8+8CCK?<K@D<NFLC;9<I<;LE;8EK 4FL8CI<8;P know @K?8JGIFG<IKP+JFK?<GIFG<IKPPFLcI< investigating @J,e<M<EK?FL>?

 PFLcI<CFFB@E>8KK?<J@Q<F=>IFLG,+N@K?@E>IFLG+EFKK?<J@Q<F=>IFLG,N@K?@E>IFLG+hN?@:?NFLC;9<EFEJ<EJ<i /?@J@J

 N?8K@KD<8EJKFK8B<K?<GIFG<IKPFEK?<I@>?Ke?8E;J@;<8J given;@KD<8EJPFLBEFNPFLcI<NFIB@E>FECPN@K?@EK?<>IFLGF=K?@E>JK?8K?8M<GIFG<IKP+ 2?<EPFL:FEJKI@:KPFLI=F:LJF=8KK<EK@FEKFJ<<FECPK?@JJD8CC<I>IFLGD8EPFK?<IGIF989@C@K@<J:?8E>< $= 

 PFLcI<K8B@E>+8J given, K?<EGh,+i<HL8CJALJKGh,ie8KC<8JK relative to the group P. /?< old Gh,iK?<=I<HL<E:PF=`K?@E>JK?8K?8M<GIFG<IKP,N@K?@EK?<<EK@I<J8DGC<a@JI<M@J<;KFK?<E<N =I<HL<E:PF=`K?@E>JK?8K?8M<GIFG<IKP,N@K?@EK?<JL9J8DGC<F= 

K?@E>JK?8K?8M<GIFG<IKP+a $=+@J given, @=+@JFLI<EK@I<NFIC;

 &! &,-!,!.0($&,!'&'1+,'*%

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 78/309

K?<ECFFB@E>=FI,+@JK?<J8D<8JCFFB@E>=FIALJK,

$=PFL:FEJKI@:KPFLI=F:LJF=8KK<EK@FEKFFECPK?<GFGLC8K@FEF= 

<>>JK?8K8I<G8@EK<;9CL<K?<EJL;;<ECP`K?<GIF989@C@KPK?8K8E<>>:FEK8@EJ8G<8ICa9<:FD<J8;@==<I<EKELD9<IK?@JGIFGFIK@FE@J;@==<I<EK=FIK?<GFGLC8K@FEF=9CL<<>>JK?8EK?<GFGLC8K@FEF= 8CC<>>J /?< given, K?<GIFG<IKPK?8K:FEJKI@:KJFLI=F:LJF= 8KK<EK@FE@J8CN8PJFEK?< right J@;<F=Gh,R+iK?<+9<:FD<JFLI

 NFIC;K?<<EK@I<K?@E>N<J<<8E;FEK?<FK?<IJ@;<F=K?<`>@M<Ea+8CN8PJ?8JGIF989@C@KPeK?8K@JN?8K@KD<8EJKFK8B<+8J

 >@M<E .FGh,R+iD<8EJ $=+?8JGIF989@C@KPN?8K@JK?<GIF989@C@KPF=,aFI`$=N<:FEJKI@:KFLI8KK<EK@FEKFFECPK?@E>J

FI<M<EKJN?<I<+@JKIL<N?8K@JK?<GIF989@C@KPF=,a ,FEK?<FK?<IJ@;<F=K?<>@M<E@J not :<IK8@Ee@KJGIF989@C@KPD8P9<nFInFI8EPFK?<IELD9<I .FN?<EPFLLJ<8P<Jc/?<FI<D8E;PFLNI@K<K?<G8IKFEK?<C<=KJ@;<8JGhR3ie?FNKF update K?<GIF989@C@KPF=8=K<IJ<<@E>3K?<E<NGIF989@C@KPF= given K?8K

 N<BEFN3K?<;<>I<<KFN?@:?3 implies ePFL:8EK<CCK?8K3@J8CN8PJK?< observation FIK?< evidence, 8E;@JK?<GIFG<IKP9<@E> @EM<JK@>8K<;K?<K?@E>PFLN8EKKFBEFN89FLK

/?<I@>?KJ@;<F=8P<Jc/?<FI<D@J;<I@M<;=IFDK?<C<=KJ@;<K?IFL>?K?<J<JK<GJ

DP0 DP0

DP0D0

D0

DP0

D0

D0D0R

DP0D0PD

D0PDD0PRDR

/?<=@IJKJK<G DP0 KF D0D0D8PCFFBC@B<8 K8LKFCF>P /?<8:KL8CD8K?G<I=FID<;@J;@==<I<EKK?FL>?DP0 @J8J@E>C<ELD9<IK?<EFID8C@Q<;GIF989@C@KPFI=I<HL<E:PF=N@K?@EK?<JL9>IFLG3 D0D0 8I<LJL8CCP 

K?<G<I:<EK8><=I<HL<E:@<JF=38E;3N@K?@EK?<<EK@I<

%(&,**!,'*1

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 79/309

J8DGC<9LKK?<:8C:LC8K@FE8CJFNFIBJ@=38E;38I<89JFCLK<ELD9<IJF=G<FGC<<M<EKJFIK?@E>J D86B8:FPDCG>H>J: @J8J@E>C<G<I:<EK8><=I<HL<E:PGIF989@C@KP8CN8PJ9<KN<<E8E;

DCG>H>J:86B8:FDCG>H>J: :8E9<D<8JLI<;<@K?<I@EGIF989@C@K@<JJL:?8JFI@KD@>?K9<<OGI<JJ<;@E >IFLGJF=NFD<E=FI<O8DGC< JCFE>8J9FK?K?<ELD<I8KFI8E;;<EFD@E8KFI8I<D<8JLI<;@EK?<J8D<LE@KJ@KJ?FLC;D8B<EF;@==<I<E:<

"F@E>=IFD D0 @EK?<;<EFD@E8KFIKF D0D0R @J8  M<IPJKI8@>?K=FIN8I;JK<GN?FJ<D8@EGLIGFJ<@J8J8JK<GG@E> JKFE<KFK?<C8JK<HL8K@FE #FN<M<IFE<:FDDFE8I@K?D<K@:8C

D@JK8B<@E8P<J@8E:8C:LC8K@FEJ@JKF;@M@;< D0 9P D0R@EJK<8;F=;@M@;@E> D0 9P 2D0D0R3 !FI<O8DGC<JFD<FE<;F@E>K?<9I<8JK:8E:<I:8C:LC8K@FEKI@<JKF><KK?<GFJK<I@FIGIF989@C@KP9PG<I=FID@E>K?<D8K?FG<I8K@FE @EJK<8;F=hi $C@B<KFK?@EBF=K?@J8J8IFJ<e=CFN<IJ<IIFI .FD<K@D<J@=PFLJ?FNPFLE>:?@C;I<E8G@:KLI<

 N@K?<@>?KIFJ<J8E;KNFKLC@GJK?<PcCCJ8PK?8KK?<G@:KLI<:FEK8@EJDFI<IFJ<JK?8E=CFN<IJ h/<:?E@:8CCPK?@JNFLC;9<:8CC<;8:C8JJ@E:CLJ@FE<IIFIi 4FL?8M<KF add K?<IFJ<J8E;K?<KLC@GJKF><KK?<ELD9<IF= flowersN?@:?PFLE<<;KF=@E;K?<GIFGFIK@FEF= IFJ<J within K?<=CFN<IJ 4FL:8EcK=@E;K?<GIFGFIK@FEF=IFJ<J@EK?<KLC@GJFIK?<GIFGFIK@FEF=KLC@GJ@EK?<IFJ<J 2?<EPFLCFFB8KK?<>I8G?K?<9FKKFD98I:FEJ@JKJF= all K?<G8K@<EKJN@K?GFJ@K@M<I<JLCKJ /?8KcJN?8KK?<;F:KFIJ<<Je8G8K@<EKN@K?8 GFJ@K@M<I<JLCK /?<HL<JK@FEK?<E9<:FD<JN?<K?<IK?@J@J8 ?<8CK?PG8K@<EKN@K?8GFJ@K@M<I<JLCKFI8:8E:<IFLJG8K@<EKN@K?8 GFJ@K@M<I<JLCK /F=@>LI<K?<F;;JF=K?8KPFL?8M<KFCFFB8KK?<

GIFGFIK@FEF=:8E:<IFLJG8K@<EKJN@K?GFJ@K@M<I<JLCKJN@K?@E8CCG8K@<EKJN?F?8M<GFJ@K@M<I<JLCKJ9<:8LJ<8>8@E`8G8K@<EKN@K?8 GFJ@K@M<I<JLCKa@JN?8KPFL8:KL8CCPJ<< 4FL:8EcK;@M@;<9P 9<:8LJ<K?8KNFLC;D<8EPFLN<I<KIP@E>KF=@E;K?<GIFGFIK@FEF=:8E:<IFLJG8K@<EKJN@K?GFJ@K@M<I<JLCKJN@K?@EK?<

 >IFLGF=?<8CK?PG8K@<EKJN@K?GFJ@K@M<I<JLCKJ@KcJC@B<8JB@E>?FN D8EPF=K?<KLC@GJ8I<IFJ<J@EJK<8;F=8JB@E>?FND8EPF=K?<=CFN<IJ8I<IFJ<J $D8>@E<LJ@E>K?<J8D<D<K?F;KF=@E;K?<GIFGFIK@FEF= healthy G8K@<EKJ 4FLNFLC;;@M@;<9P8E;

=@E;K?8KnF=K?<G8K@<EKJN<I<?<8CK?P *IKF9<<O8:KPFL

 &! &,-!,!.0($&,!'&'1+,'*%

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 80/309

 NFLC;=@E;K?8KnF=:8E:<IFLJG8K@<EKJN@K?GFJ@K@M<I<JLCKJ N<I<?<8CK?PG8K@<EKJN@K?GFJ@K@M<I<JLCKJ

/?<C8JKJK<G@E;<I@M@E>8P<Jc/?<FI<D@J>F@E>=IFD D0 KFD0PD@E9FK?K?<ELD<I8KFI8E;K?<;<EFD@E8KFI8E;=IFD D0R KF D0PRDR@EK?<;<EFD@E8KFI

2?P 2<CCFE<8EJN<I@J9<:8LJ<Gh3RiGh3RSi8E;Ghi:FII<JGFE;KFK?<@E@K@8C@E=FID8K@FE>@M<E@E8CCK?<JKFIP GIF9C<DJ LKN?PN<I<K?<JKFIPGIF9C<DJNI@KK<EK?8KN8P

<:8LJ<@ED8EP:8J<JGh3RiGh3RSi8E;Ghi8I<N?8KN<

8:KL8CCP know; 8E;K?@J@EKLIE?8GG<EJ9<:8LJ<Gh3Ri8E;Gh3RSi8I<F=K<EK?<HL8EK@K@<JK?8K;@I<:KCP;<J:I@9< causal relations, N@K?K?<FK?<IHL8EK@K@<J;<I@M<;=IFDK?<D8E;Ghi8J statistical  relations. !FI<O8DGC<Gh3RiK?<@DGC@:8K@FE=IFDKF3N?<I< @JN?8KN<N8EKKFBEFN8E;3@JFLIN8PF=F9J<IM@E>@K:FII<JGFE;JKFK?<@DGC@:8K@FE=IFD8NFD8E?8M@E>9I<8JK:8E:<IKF8GFJ@K@M<D8DDF>I8G?P /?@J@JEFKALJK8 statistical implication9LK8 direct causal relation; 8NFD8E><KJ8GFJ@K@M<D8DDF>I8G?P because J?<?8J9I<8JK:8E:<I /?<D8DDF>I8G?P@J designed KF;<K<:K9I<8JK:8E:<I8E;@K@J8=8:K89FLKK?<G?PJ@:8CGIF:<JJF= K?<D8DDF>I8G?P<O8DK?8K@K?8J8EnGIF989@C@KPF= ;<K<:K@E>9I<8JK:8E:<I JCFE>8JK?<;<J@>EF=K?<D8DDF>I8G?PD8:?@E<JK8PJ:FEJK8EKGh3RiN@CCJK8P8Kn<M<E@=Ghi:?8E><Je=FI<O8DGC<@=N<J:I<<E8>IFLGF=NFD8E

 N@K?FK?<II@JB=8:KFIJJFK?8KK?<GI@FI=I<HL<E:PF=NFD<EN@K?9I<8JK:8E:<I@Jn@EJK<8;F=n $EK?@J:8J<Gh3iN@CC:?8E><8CFE>N@K?Ghi8E;JFN@CCGh3iGhR3i8E;JFFE9LK

Gh3RiJK8PJ8Kn9<:8LJ<K?8KcJ8=8:K89FLKK?<D8DDF>I8G?P <O8D@KJ<C= h/?FL>?PFL;FE<<;KFK<JKK?@JJK8K<D<EK9<=FI<I<CP@E>FE@K@KcJGFJJ@9C<K?8KK?<D8DDF>I8G?P<O8DD@>?KNFIB9<KK<IFEJFD<=FIDJF=9I<8JK:8E:<IK?8EFK?<IJi Gh3Ri@JFE<F=K?< simple =8:KJ=IFDN?@:?:FDGC<O=8:KJC@B<Gh3i8I<:FEJKIL:K<;Gh3Ri@J8E elementary :8LJ8CI<C8K@FEN@K?@E8 :FDGC<OJPJK<D8E;@K?8J8;@I<:KG?PJ@:8C@EK<IGI<K8K@FE /?@J@J

 N?P8P<Jc/?<FI<D?8JK?<=FID@K;F<J@KcJEFK=FIJFCM@E>D8K?9I8@EK<8J<IJ9LK=FII<8JFE@E>89FLKK?<G?PJ@:8CLE@M<IJ<

%(&,**!,'*1

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 81/309

*E:<K?<;<I@M8K@FE@J=@E@J?<;8CCK?<@DGC@:8K@FEJFEK?<I@>?KJ@;<F=K?<<HL8K@FE8I<F=K?<=FID D0P FI D0PRN?@C<K?<@DGC@:8K@FEFEK?<C<=KJ@;<@J DP0 JCFE>8JPFL

I<D<D9<IK?@J8E;PFL><KK?<I<JKF=K?<<HL8K@FEI@>?K@KJ?FLC;EcKD8KK<IN?<K?<IPFL?8GG<E<;KFJK8IKFLKN@K?GhR3iFIGh3RiFEK?<C<=KJ@;<F=K?<<HL8K@FE8JCFE>8JK?<ILC<J8I<8GGC@<; consistently e@=PFLJK8IK<;FLKN@K?K?<;@I<:K@FEF= @DGC@:8K@FEGh3RiFEK?<C<=KJ@;<F=K?<<HL8K@FEPFLNFLC;E<<;KF<E;LGN@K?K?<;@I<:K@FEGhR3iFEK?<I@>?KJ@;<F=K?<<HL8K@FE /?@JF=:FLIJ<@JALJK:?8E>@E>K?<M8I@89C<C89<CJK?<GF@EK@JKFI<D<D9<IK?<JPDD<KIP@EFI;<IKFI<D<D9<IK?<JKIL:KLI<F=8P<Jc/?<FI<D

/?<JPDD<KIP8I@J<J9<:8LJ<K?<<C<D<EK8IP causal relations 8I< ><E<I8CCP@DGC@:8K@FEJ=IFD=8:KJKFF9J<IM8K@FEJ@<=IFD9I<8JK:8E:<IKFGFJ@K@M<D8DDF>I8G?P /?<<C<D<EK8IP steps in reasoning 8I<><E<I8CCP@DGC@:8K@FEJ=IFDF9J<IM8K@FEJKF=8:KJ@<=IFD8 GFJ@K@M<D8DDF>I8G?PKF9I<8JK:8E:<I /?<C<=KJ@;<F=8P<Jc/?<FI<D@J8E<C<D<EK8IP inferential JK<G=IFDK?<F9J<IM8K@FEF= GFJ@K@M<D8DDF>I8G?PKFK?<:FE:CLJ@FEF=8E@E:I<8J<;GIF989@C@KPF=9I<8JK:8E:<I $DGC@:8K@FE@JNI@KK<EI@>?KeKFeC<=KJF

 N<NI@K< D86B8:FPDCG>H>J: FEK?<C<=KJ@;<F=K?<<HL8K@FE /?<I@>?KJ@;<F=8P<Jc/?<FI<D;<J:I@9<JK?<<C<D<EK8IP causal JK<GJe=FI<O8DGC<=IFD9I<8JK:8E:<IKF8 GFJ@K@M<D8DDF>I8G?Pe8E;JFK?<@DGC@:8K@FEJFEK?<I@>?KJ@;<F=8P<Jc/?<FI<DK8B<K?<=FID DDCG>H>J:P86B8:F FIDDCG>H>J:PR86B8:F

 E;K?8KcJ8P<Jc/?<FI<D -8K@FE8C@E=<I<E:<FEK?<C<=K<E;

G?PJ@:8C:8LJ8C@KPFEK?<I@>?K<E;8E<HL8K@FEN@K?D@E;FEFE<J@;<8E;I<8C@KPFEK?<FK?<I -<D<D9<I?FNK?<J:@<EK@=@:D<K?F;KLIE<;FLKKF9<8JG<:@8C:8J<F=8P<Jc/?<FI<D $=PFL

 N8EK<;KFGLK@KGF<K@:8CCPPFL:FLC;J8PK?8K8P<Jc/?<FI<D9@E;JI<8JFE@E>@EKFK?<G?PJ@:8CLE@M<IJ<

*B8PN<cI<;FE<

Reverend Bayes says:

 &! &,-!,!.0($&,!'&'1+,'*%

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 82/309

 You are now an initiateof the Bayesian Conspiracy.

%(&,**!,'*1

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 83/309

4. Why truth? And…

.FD< F= K?< :FDD<EKJ @E K?@J 9CF> ?8M< KFL:?<; FE K?< HL<JK@FE

F= N?P N< FL>?K KF J<<B KILK? h/?8EB=LCCP EFK D8EP ?8M< HL<JeK@FE<; N?8K KILK? @Ji *LI J?8G@E> DFK@M8K@FE =FI :FE=@>LI@E> FLIK?FL>?KJ KF I8K@FE8C@KP N?@:? ;<K<ID@E<J N?<K?<I 8 >@M<E :FE=@>eLI8K@FE @J `>FF;a FI `98;a :FD<J =IFD N?P<M<I N< N8EK<; KF =@E;KILK?@EK?<=@IJKGC8:<

$K @J NI@KK<E `/?< =@IJK M@IKL< @J :LI@FJ@KPa LI@FJ@KP @J FE< I<8eJFE KF J<<B KILK? 8E; @K D8P EFK 9< K?< FECP FE< 9LK @K ?8J 8 JG<:@8C8E; 8;D@I89C< GLI@KP $= PFLI DFK@M< @J :LI@FJ@KP PFL N@CC 8JJ@>E

GI@FI@KP KF HL<JK@FEJ 8::FI;@E> KF ?FN K?< HL<JK@FEJ K?<DJ<CM<JK@:BC< PFLI G<IJFE8C 8<JK?<K@: J<EJ< KI@:B@<I :?8CC<E>< N@K? 8  >I<8K<I GIF989@C@KP F= =8@CLI< D8P 9< NFIK? DFI< <==FIK K?8E 8 J@DeGC<IFE<ALJK9<:8LJ<@K@JDFI<=LE

.FD< G<FGC< $ JLJG<:K D8P F9A<:K K?8K :LI@FJ@KP @J 8E <DFK@FE8E; @J K?<I<=FI< `EFK I8K@FE8Ca $ C89<C 8E <DFK@FE 8J `EFK I8K@FE8Ca@= @K I<JKJ FE D@JK8B<E 9<C@<=J FI I8K?<I FE @II8K@FE8C <G@JK<D@: :FEe;L:K `$= K?< @IFE 8GGIF8:?<J PFLI =8:< 8E; PFL 9<C@<M< @K @J ?FK8E; @K @J :FFC K?< 28P FGGFJ<J PFLI =<8I $= K?< @IFE 8GGIF8:?<J

 PFLI =8:< 8E; PFL 9<C@<M< @K @J :FFC 8E; @K @J ?FK K?< 28P FGeGFJ<J PFLI :8CDa FEM<IJ<CP K?<E 8E <DFK@FE N?@:? @J <MFB<;9P :FII<:K 9<C@<=J FI <G@JK<D@:8CCP I8K@FE8C K?@EB@E> @J 8 `I8K@FE8C<DFK@FEa 8E; K?@J ?8J K?< 8;M8EK8>< F= C<KK@E> LJ I<>8I; :8CD 8J8E <DFK@FE8C JK8K< I8K?<I K?8E 8 GI@M@C<><; ;<=8LCK 2?<E G<FGC<K?@EB F= `<DFK@FEa 8E; `I8K@FE8C@KPa 8J FGGFJ<; $ JLJG<:K K?8K K?<P 8I< I<8CCP K?@EB@E> F= .PJK<D 8E; .PJK<D g=8JK G<I:<GKL8C AL;>eD<EKJ M<IJLJ JCFN ;<C@9<I8K@M< AL;>D<EKJ <C@9<I8K@M< AL;>D<EKJ8I<EcK 8CN8PJ KIL< 8E; G<I:<GKL8C AL;>D<EKJ 8I<EcK 8CN8PJ =8CJ< JF

@K @J M<IP @DGFIK8EK KF ;@JK@E>L@J? K?8K ;@:?FKFDP =IFD `I8K@FE8C@eKPa FK? JPJK<DJ :8E J<IM< K?< >F8C F= KILK? FI ;<=<8K @K 8::FI;@E> KF?FNK?<P8I<LJ<;

<J@;<J J?<<I <DFK@FE8C :LI@FJ@KP N?8K FK?<I DFK@M<J 8I< K?<I<=FI ;<J@I@E> KILK? 2<CC PFL D@>?K N8EK KF 8::FDGC@J? JFD< JG<e:@=@: I<8CeNFIC; >F8C C@B< 9L@C;@E> 8E 8@IGC8E< 8E; K?<I<=FI< PFLE<<; KF BEFN JFD< JG<:@=@: KILK? 89FLK 8<IF;PE8D@:J *I DFI<DLE;8E<CP PFL N8EK :?F:FC8K< D@CB 8E; K?<I<=FI< PFL N8EK KF

http://lesswrong.com/lw/go/why_truth_and/  http://sl4.org/wiki/TheSimpleTruth

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 84/309

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 85/309

$ K<E; KF 9< JLJG@:@FLJ F= DFI8C@KP 8J 8 DFK@M8K@FE =FI I8K@FeE8C@KP not  9<:8LJ< $ I<A<:K K?< DFI8C @;<8C 9LK 9<:8LJ< @K @EM@K<J:<IK8@E B@E;J F= KIFL9C< $K @J KFF <8JP KF 8:HL@I< 8J C<8IE<; DFI8C

;LK@<J DF;<J F= K?@EB@E> K?8K 8I< ;I<8;=LC D@JJK<GJ @E K?< ;8E:<FEJ@;<I (I .GF:B F= Star Trek 8 E8@M< 8I:?<KPG< F= I8K@[email protected]:BcJ <DFK@FE8C JK8K< @J 8CN8PJ J<K KF `:8CDa <M<E N?<E N@C;CP @E8GGIFGI@8K< #< F=K<E >@M<J D8EP J@>E@=@:8EK ;@>@KJ =FI GIF989@Ce@K@<J K?8K 8I< >IFJJCP LE:8C@9I8K<; h> `8GK8@E @= PFL JK<<I K?<EK<IGI@J< ;@I<:KCP @EKF K?8K 9C8:B ?FC< FLI GIF989@C@KP F= JLIM@M@E> @J FECP na 4<K E@E< K@D<J FLK F= K<E K?< EK<IGI@J< @J EFK ;<eJKIFP<; 2?8K B@E; F= KI8>@: =FFC >@M<J =FLI J@>E@=@:8EK ;@>@KJ =FI 8 =@>LI< K?8K @J F== 9P KNF FI;<IJ F= D8>E@KL;<i 4<K K?@J GFGLC8I @De

8>< @J ?FN D8EP G<FGC< :FE:<@M< F= K?< ;LKP KF 9< `I8K@FE8CagJD8CC NFE;<I K?8K K?<P ;F EFK <D9I8:< @K N?FC<?<8IK<;CP /F D8B< I8eK@FE8C@KP @EKF 8 DFI8C ;LKP @J KF >@M< @K 8CC K?< ;I<8;=LC ;<>I<<J F= =I<<;FD F= 8E 8I9@KI8IP KI@98C :LJKFD +<FGC< 8II@M< 8K K?< NIFE> 8EJN<I 8E; K?<E @E;@>E8EKCP GIFK<JK K?8K K?<P 8:K<; N@K? GIFGI@<KPI8K?<IK?8EC<8IE@E>=IFDK?<@ID@JK8B<

 E; P<K @= N<cI< >F@E> KF improve FLI JB@CCJ F= I8K@FE8C@KP >F 9<e PFE; K?< JK8E;8I;J F= G<I=FID8E:< J<K 9P ?LEK<Ie>8K?<I<IJ N<cCCE<<; ;<C@9<I8K< 9<C@<=J 89FLK ?FN KF K?@EB N@K? GIFGI@<KP 2?<E

 N< NI@K< E<N D<EK8C GIF>I8DJ =FI FLIJ<CM<J K?<P JK8IK FLK @E .PJeK<D K?< ;<C@9<I8K< JPJK<D 8E; 8I< FECP JCFNCPg@= <M<IgKI8@E<;@EKF K?< E<LI8C :@I:L@KIP K?8K LE;<IC@<J .PJK<D .F @= K?<I< 8I<:<IK8@E B@E;J F= K?@EB@E> K?8K N< =@E; N< N8EK KF avoid  gC@B< J8P9@8J<Jg@K N@CC <E; LG I<GI<J<EK<; N@K?@E .PJK<D 8J 8E @EALE:K@FEEFKKFK?@EBK?8KN8P8GIF=<JJ<;;LKPF=8MF@;8E:<

$= N< N8EK K?< KILK? N< :8E DFJK <==<:K@M<CP F9K8@E @K 9P K?@EBe@E> @E :<IK8@E N8PJ I8K?<I K?8E FK?<IJ 8E; K?<J< 8I< K?< K<:?E@HL<J

F= I8K@FE8C@KP .FD< F= K?< K<:?E@HL<J F= I8K@FE8C@KP @EMFCM< FM<Ie:FD@E>8:<IK8@E:C8JJF=F9JK8:C<JK?<9@8J<J_

hFEK@EL<;@EE<OKGFJK `2?8KcJ89@8J8>8@Eai

/1,*-, &3

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 86/309

5. What is Evidence?

`/?<J<EK<E:<bJEFN@JN?@K<c@J true @=8E;FECP@=JEFN@J

 N?@K<a gC=I<;/8IJB@

`/FJ8PF=N?8K@JK?8K@K@JFIF=N?8K@JEFKK?8K@K@JEFK@J true.a

 gI@JKFKC< Metaphysics IV 

$= K?<J< KNF HLFK<J ;FEcK J<<D C@B< 8 JL==@:@<EK ;<=@E@K@FE F= `KILK?aI<8; K?@J /F;8P $cD >F@E> KF K8CB 89FLK `<M@;<E:<a h$ 8CJF @EK<E;KF ;@J:LJJ 9<C@<=JeF=e=8:K EFK <DFK@FEJ FI DFI8C@KP 8J ;@JK@E>L@J?<;?<I<i

28CB@E> 8CFE> K?< JKI<<K PFLI J?F<C8:<J :FD< LEK@<; .?FIKCP K?<I<8=K<I =FI JFD< F;; I<8JFE PFL JK8IK believing PFLI J?F<C8:<J8I< LEK@<; '@>?K C<8M<J K?< .LE 8E; JKI@B<J PFLI J?F<C8:<J 8E;9FLE:<J F== JFD< G?FKFEJ <EK<I K?< GLG@CJ F= PFLI <P<J 8E; JKI@B<

 PFLI I<K@E8 K?< <E<I>P F= K?< G?FKFEJ KI@>><IJ E<LI8C @DGLCJ<J K?<E<LI8C @DGLCJ<J 8I< KI8EJD@KK<; KF K?< M@JL8CeGIF:<JJ@E> 8I<8J F= K?<9I8@E 8E; K?<I< K?< FGK@:8C @E=FID8K@FE @J GIF:<JJ<; 8E; I<:FEe

JKIL:K<; @EKF 8 DF;<C K?8K @J I<:F>E@Q<; 8J 8E LEK@<; J?F<C8:</?<I< @J 8 J<HL<E:< F= <M<EKJ 8 :?8@E F= :8LJ< 8E; <==<:K N@K?@EK?< NFIC; 8E; PFLI 9I8@E 9P N?@:? PFL <E; LG 9<C@<M@E> N?8K PFL9<C@<M< /?< =@E8C FLK:FD< F= K?< GIF:<JJ @J 8 JK8K< F= mind N?@:?D@IIFIJK?<JK8K<F=PFLI8:KL8C shoelaces

2?8K @J evidence?  $K @J 8E <M<EK <EK8E>C<; 9P C@EBJ F= :8LJ<8E; <==<:K N@K? N?8K<M<I PFL N8EK KF BEFN 89FLK $= K?< K8I><K F= 

 PFLI @EHL@IP @J PFLI J?F<C8:<J =FI <O8DGC< K?<E K?< C@>?K <EK<I@E>  PFLI GLG@CJ @J <M@;<E:< <EK8E>C<; N@K? PFLI J?F<C8:<J /?@J J?FLC;

EFK 9< :FE=LJ<; N@K? K?< K<:?E@:8C J<EJ< F= `<EK8E>C<D<EKa LJ<; @EG?PJ@:Jg?<I< $cD ALJK K8CB@E> 89FLK `<EK8E>C<D<EKa @E K?< J<EJ< F= KNF K?@E>J K?8K <E; LG @E :FII<C8K<; JK8K<J 9<:8LJ< F= K?< C@EBJ F= :8LJ<8E;<==<:K9<KN<<EK?<D

)FK <M<IP @E=CL<E:< :I<8K<J K?< B@E; F= `<EK8E>C<D<EKa I<HL@I<;=FI <M@;<E:< $KcJ EF ?<CG KF ?8M< 8 D8:?@E< K?8K 9<<GJ N?<E PFL<EK<I N@EE@E> CFKK<IP ELD9<IJ @= K?< D8:?@E< also 9<<GJ N?<E PFL

http://lesswrong.com/lw/jl/what_is_evidence/ 

http://sl4.org/wiki/TheSimpleTruth +8><b!<<C@E>-8K@FE8Cc

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 87/309

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 88/309

<==<:K 2?<E PFL J8P `(P J?F<C8:<J 8I< LEK@<;a FM<I 8 :<CCG?FE< PFLcI< J?8I@E> PFLI <EK8E>C<D<EK N@K? PFLI J?F<C8:<J N@K? 8 =I@<E;

/?<I<=FI< I8K@FE8C 9<C@<=J 8I< :FEK8>@FLJ 8DFE> ?FE<JK =FCB

 N?F 9<C@<M< <8:? FK?<I KF 9< ?FE<JK E; @KcJ N?P 8 :C8@D K?8K PFLI 9<C@<=J 8I< not :FEK8>@FLJgK?8K PFL 9<C@<M< =FI GI@M8K< I<8JFEJ N?@:? 8I< EFK KI8EJD@JJ@9C<g@J JF JLJG@:@FLJ $= PFLI 9<C@<=J 8I< <EeK8E>C<;N@K?I<8C@KPK?<P should 9<:FEK8>@FLJ8DFE>?FE<JK=FCB

$= PFLI DF;<C F= I<8C@KP JL>><JKJ K?8K K?< FLKGLKJ F= PFLIK?FL>?K GIF:<JJ<J J?FLC; not  9< :FEK8>@FLJ KF FK?<IJ K?<E PFLIDF;<C J8PJ K?8K PFLI 9<C@<=J 8I< EFK K?<DJ<CM<J <M@;<E:< D<8E@E> K?<P 8I< EFK <EK8E>C<; N@K? I<8C@KP 4FL J?FLC; 8GGCP 8 I<=C<:K@M<:FII<:K@FE8E;JKFG9<C@<M@E>

$E;<<; @= PFL feel, FE 8  gut  C<M<C N?8K K?@J 8CC means, PFL N@CC automatically JKFG 9<C@<M@E> <:8LJ< `DP 9<C@<= @J EFK <EK8E>C<; N@K? I<8C@KPa means `DP 9<C@<= @J EFK 8::LI8K<a J JFFE 8J PFL JKFG9<C@<M@E> `bJEFN @J N?@K<c @J KIL<a PFL J?FLC; h8LKFD8K@:8CCPi JKFG9<C@<M@E>`JEFN@JN?@K<aFIJFD<K?@E>@JM<IPNIFE>

.F >F 8?<8; 8E; <OGC8@E N?P K?< B@E; F= K?FL>?K GIF:<JJ<J PFLLJ< JPJK<D8K@:8CCP GIF;L:< 9<C@<=J K?8K D@IIFI I<8C@KP OGC8@E N?P 

 PFL K?@EB PFLcI< rational. 2?P PFL K?@EB K?8K LJ@E> K?FL>?K GIFe

:<JJ<J C@B< K?< FE<J PFL LJ< D@E;J N@CC <E; LG 9<C@<M@E> `JEFN @J N?@K<a @= 8E; FECP @= JEFN @J N?@K< $= PFL ;FEcK 9<C@<M< K?8K K?<FLKGLKJ F= PFLI K?FL>?K GIF:<JJ<J 8I< <EK8E>C<; N@K? I<8C@KP N?P ;F PFL 9<C@<M< K?< FLKGLKJ F= PFLI K?FL>?K GIF:<JJ<J $KcJ K?< J8D<K?@E>FI@KJ?FLC;9<

%(&,**!,'*1

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 89/309

6. How Much Evidence Does It Take?

Followup to: 2?8K@JM@;<E:<

+I<M@FLJCP  $ ;<=@E<; evidence 8J `8E <M<EK <EK8E>C<; 9P C@EBJ F= :8LJ< 8E; <==<:K N@K? N?8K<M<I PFL N8EK KF BEFN 89FLKa 8E; en-tangled 8J `?8GG<E@E> ;@==<I<EKCP =FI ;@==<I<EK GFJJ@9C< JK8K<J F= K?<K8I><Ka .F ?FN DL:? <EK8E>C<D<EKg?FN DL:? <M@;<E:<g@J I<eHL@I<;KFJLGGFIK89<C@<=

'<KcJ JK8IK N@K? 8 HL<JK@FE J@DGC< <EFL>? KF 9< D8K?<D8K@:8C?FN ?8I; NFLC; PFL ?8M< KF <EK8E>C< PFLIJ<C= N@K? K?< CFKK<IP 

@E FI;<I KF N@E .LGGFJ< K?<I< 8I< J<M<EKP 98CCJ ;I8NE N@K?FLK

I<GC8:<D<EK 8E; J@O ELD9<IJ KF D8K:? =FI K?< N@E /?<E K?<I<8I< GFJJ@9C< N@EE@E> :FD9@E8K@FEJ ?<E:< 8 I8E;FDCP J<C<:K<; K@:B<K NFLC; ?8M< 8 GIF989@C@KP F= N@EE@E> hni /F N@E K?< CFKK<IP PFL NFLC; E<<; <M@;<E:< se- lective <EFL>? KF M@J@9CP =8MFI FE< :FD9@E8K@FE FM<I 8CK<IE8K@M<J

.LGGFJ< K?<I< 8I< JFD< K<JKJ PFL :8E G<I=FID N?@:? ;@J:I@De@E8K< GIF989@C@JK@:8CCP 9<KN<<E N@EE@E> 8E; CFJ@E> CFKK<IP ELDe9<IJ !FI <O8DGC< PFL :8E GLE:? 8 :FD9@E8K@FE @EKF 8 C@KKC< 9C8:B

9FO K?8K 8CN8PJ 9<<GJ @= K?< :FD9@E8K@FE @J K?< N@EE<I 8E; ?8JFECP 8 hni :?8E:< F= 9<<G@E> @= K?< :FD9@E8K@FE @J NIFE>$E 8P<J@8E K<IDJ N< NFLC; J8P K?< likelihood ratio @J KF /?@JD<8EJ K?8K K?< 9FO @J K@D<J 8J C@B<CP KF 9<<G N?<E N< GLE:? @E8 :FII<:K :FD9@E8K@FE :FDG8I<; KF ?FN C@B<CP @K @J KF 9<<G =FI 8E@E:FII<:K:FD9@E8K@FE

/?<I< 8I< JK@CC 8 N?FC< CFK F= GFJJ@9C< :FD9@E8K@FEJ $= PFLGLE:? @E @E:FII<:K :FD9@E8K@FEJ K?< 9FO N@CC 9<<G FE F= K?<D

9P J?<<I :?8E:< hFE 8M<I8><i $= PFL GLE:? @E 8CC GFJJ@e9C< :FD9@E8K@FEJ K?<E N?@C< K?< 9FO @J :<IK8@E KF 9<<G =FI K?< FE< N@EE@E> :FD9@E8K@FE @K N@CC 8CJF 9<<G =FI CFJ@E> :FD9@eE8K@FEJhFE8M<I8><i

.F K?@J 9FO ;F<JEcK C<K PFL N@E K?< CFKK<IP 9LK @KcJ 9<KK<I K?8EEFK?@E> $= PFL LJ<; K?< 9FO PFLI F;;J F= N@EE@E> NFLC; >F =IFD

http://lesswrong.com/lw/jn/how_much_evidence_does_it_take/  +8><b2?8K@JM@;<E:<c +8><b2?8K@JM@;<E:<c

http://lesswrong.com/lw/hl/lotteries_a_waste_of_hope/  +8><bE$EKL@K@M<OGC8E8K@FEF=8P<J/?<FI<Dc

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 90/309

@E KF @E 4FLcM< D8;< JFD< GIF>I<JJ KFN8I;=@E;@E> PFLI K8I><K K?< KILK? N@K?@E K?< ?L>< JG8:< F= GFJJ@9@C@K@<J

.LGGFJ< PFL :8E LJ< 8EFK?<I 9C8:B 9FO KF K<JK :FD9@E8K@FEJ

twice, independently. FK? 9FO<J 8I< :<IK8@E KF 9<<G =FI K?< N@EE@E> K@:B<K LK K?< :?8E:< F= 8 9FO 9<<G@E> =FI 8 CFJ@E> :FD9@E8K@FE @J independently =FI <8:? 9FO ?<E:< K?< :?8E:< F= both 9FO<J 9<<Ge@E> =FI 8 CFJ@E> :FD9@E8K@FE @J 2< :8E J8P K?8K K?< cumulative<M@;<E:< F= KNF @E;<G<E;<EK K<JKJ ?8J 8 C@B<C@?FF; I8K@F F= /?< ELD9<I F= CFJ@E> CFKK<IP K@:B<KJ K?8K G8JJ 9FK? K<JKJ N@CC 9< hFE8M<I8><i

.@E:< K?<I< 8I< GFJJ@9C< CFKK<IP K@:B<KJ PFL D@>?K >L<JJ K?8K PFL E<<; <M@;<E:< N?FJ< JKI<E>K? @J 8IFLE; KF

g8E <M<EK FI J<I@<J F= <M<EKJ N?@:? @J K@D<J DFI< C@B<eCP KF ?8GG<E =FI 8 N@EE@E> :FD9@E8K@FE K?8E 8 CFJ@E> :FD9@E8K@FE

 :KL8CCP K?@J 8DFLEK F= <M@;<E:< NFLC; FECP 9< <EFL>? KF >@M< PFL8E even :?8E:< F= N@EE@E> K?< CFKK<IP 2?P <:8LJ< @= PFL 8GGCP 8 =@CK<I F= K?8K GFN<I KF D@CC@FE CFJ@E> K@:B<KJ K?<I< N@CC 9< FE8M<I8>< FE< CFJ@E> K@:B<K K?8K G8JJ<J K?< =@CK<I /?< N@EE@E> K@:B<K

 N@CC 8CJF G8JJ K?< =@CK<I .F PFLcCC 9< C<=K N@K? KNF K@:B<KJ K?8K G8JJ<;K?< =@CK<I FECP FE< F= K?<D 8 N@EE<I n F;;J F= N@EE@E> @= PFL:8EFECP9LPFE<K@:B<K

  9<KK<I N8P F= M@<N@E> K?< GIF9C<D $E K?< 9<>@EE@E> K?<I< @J N@EE@E> K@:B<K 8E; CFJ@E> K@:B<KJ JF PFLI F;;J F= N@EeE@E> 8I< $= PFL LJ< 8 J@E>C< 9FO K?< F;;J F= @K 9<<G@E> 8I< =FI 8 N@EE@E> K@:B<K 8E; =FI 8 CFJ@E> K@:B<K .F N< DLCK@eGCP 9P 8E; ><K ;;@E> 8EFK?<I 9FO F= <M@;<E:< DLCK@GC@<J K?< F;;J 9P 8>8@E JF EFN K?< F;;J 8I<N@EE@E>K@:B<KKFCFJ@E>K@:B<KJ

$K @J :FEM<E@<EK KF D<8JLI< <M@;<E:< @E 9@KJgEFK C@B< 9@KJ FE

8 ?8I; ;I@M< 9LK D8K?<D8K@:@8EcJ 9@KJ N?@:? 8I< :FE:<GKL8CCP ;@=e=<I<EK (8K?<D8K@:@8EcJ 9@KJ 8I< K?< CF>8I@K?DJ 98J< F= GIF9e89@C@K@<J !FI <O8DGC< @= K?<I< 8I< =FLI GFJJ@9C< FLK:FD<J 8E; N?FJ< GIF989@C@K@<J 8I< n n n 8E; n 8E; $K<CC PFL K?< FLK:FD< N8J `a K?<E $ ?8M< KI8EJD@KK<; K?I<< 9@KJ F= @E=FID8K@FE KF PFL 9<:8LJ< $ @E=FID<; PFL F= 8E FLK:FD< N?FJ<GIF989@C@KPN8J

$K JF ?8GG<EJ K?8K @J JC@>?KCP C<JJ K?8E KF K?< K?GFN<I .F 9FO<J FI 9@KJ F= <M@;<E:<g8E <M<EK

%(&,**!,'*1

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 91/309

K@D<J DFI< C@B<CP KF ?8GG<E @= K?< K@:B<Ke?PGFK?<J@J @J KIL< K?8E@= @K @J =8CJ<gNFLC; J?@=K K?< F;;J =IFD KF N?@:? I<;L:<J KF *;;J F= KF D<8E

KNF :?8E:<J KF N@E =FI <8:? :?8E:< KF CFJ< JF K?< probability F= N@EeE@E> N@K? 9@KJ F= <M@;<E:< @J ;;@E> 8EFK?<I 9FO 8EFK?<I 9@KJ F= <M@;<E:< NFLC; K8B< K?< F;;J KF ;;@E> P<K 8EFK?<IKNF9FO<JNFLC;K8B<K?<:?8E:<F=N@EE@E>KF

.F @= PFL N8EK KF C@:<EJ< 8  strong belief  K?8K PFL N@CC N@E K?<CFKK<IPg8I9@KI8I@CP ;<=@E<; 8J C<JJ K?8E 8 n GIF989@C@KP F= 9<@E> 

 NIFE>g 9@KJ F= <M@;<E:< 89FLK K?< N@EE@E> :FD9@E8K@FE J?FLC;;FK?<KI@:B

$E ><E<I8C K?< ILC<J =FI N<@>?@E> `?FN DL:? <M@;<E:< @K K8B<Ja

=FCCFN 8 J@D@C8I G8KK<IE /?< C8I><I K?< space of possibilities @E N?@:?K?< ?PGFK?<J@J C@<J FI K?< DFI< LEC@B<CP K?< ?PGFK?<J@J J<<DJ a pri-ori  :FDG8I<; KF @KJ E<@>?9FIJ FI K?< DFI< :FE=@;<EK PFL N@J? KF9<K?<DFI<<M@;<E:<PFLE<<;

4FL :8EEFK ;<=P K?< ILC<J PFL :8EEFK =FID 8::LI8K< 9<C@<=J98J<; FE @E8;<HL8K< <M@;<E:< '<KcJ J8P PFLcM< >FK 9FO<J C@E<;LG @E 8 IFN 8E; PFL JK8IK GLE:?@E> :FD9@E8K@FEJ @EKF K?< 9FO<J4FL :8EEFK JKFG FE K?< =@IJK :FD9@E8K@FE K?8K ><KJ 9<<GJ =IFD

8CC 9FO<J J8P@E> `LK K?< F;;J F= K?8K ?8GG<E@E> =FI 8 CFJ@E> :FD9@E8K@FE 8I< 8 D@CC@FE KF FE< $cCC ALJK @>EFI< K?FJ< @MFIPeKFN<I8P<J@8E ILC<J 8E; JKFG ?<I<a *E 8M<I8>< CFJ@E> K@:B<KJ N@CCG8JJ JL:? 8 K<JK =FI <M<IP N@EE<I FEJ@;<I@E> K?< JG8:< F= GFJeJ@9@C@K@<J 8E; K?< GI@FI @DGIF989@C@KP PFL ALDG<; KF 8 KFFeJKIFE> :FE:CLJ@FE 98J<; FE @EJL==@:@<EK <M@;<E:< /?8KcJ EFK 8 GF@EKC<JJ9LI<8L:I8K@:I<>LC8K@FE@KcJD8K?

*= :FLIJ< PFL :8E JK@CC believe 98J<; FE @E8;<HL8K< <M@;<E:< @= K?8K @J PFLI N?@D 9LK PFL N@CC EFK 9< 89C< KF 9<C@<M< accurately. $K

@J C@B< KIP@E> KF ;I@M< PFLI :8I N@K?FLK 8EP =L<C 9<:8LJ< PFL ;FEcK9<C@<M< @E K?< J@CCPe;@CCP =L;;Pe;L;;P :FE:<GK K?8K @K FL>?K KF K8B<=L<C KF >F GC8:<J $K NFLC; 9< JF DL:? DFI< fun, 8E; JF DL:? C<JJ<OG<EJ@M< @= N< ALJK ;<:@;<; KF I<G<8C K?< C8N K?8K :8IJ E<<; =L<C$JEcK @K ALJK F9M@FLJCP 9<KK<I =FI <M<IPFE< 2<CC PFL :8E KIP @= K?8K@J PFLI N?@D 4FL :8E <M<E J?LK PFLI <P<J 8E; GI<K<E; K?< :8I @JDFM@E> LK KF really 8II@M< 8K 8::LI8K< 9<C@<=J I<HL@I<J <M@;<E:<e=Le<C8E;K?<=LIK?<IPFLN8EKKF>FK?<DFI<=L<CPFLE<<;

'/%-.!&'+!,,#

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 92/309

7. How to Convince Me That 2 + 2 = 3

$E`2?8K@JM@;<E:<a$ NIFK<

/?@J@JN?PI8K@FE8C@JKJGLKJL:?8?<8MPGI<D@LDFEK?<G8I8;FO@:8CeJ<<D@E>:C8@DK?8K89<C@<=@JFECPI<8CCP worthwhile @=PFL:FLC;@EGI@E:@GC<9<G<IJL8;<;KF9<C@<M<FK?<IN@J< $=PFLII<K@E8<E;<;LG@EK?<J8D<JK8K<I<>8I;C<JJF=N?8KC@>?K<EK<I<;@KPFLNFLC;9<9C@E;_ #<E:<K?<G?I8J<`9C@E;=8@K?a $=N?8KPFL9<C@<M<;F<JEcK;<G<E;FEN?8KPFLJ<<PFLcM<9<<E9C@E;<;8J<==<:K@M<CP8J9PGFB@E>FLKPFLI<P<98CCJ

@?8E8I8E I<GC@<;

$:8EEFK:FE:<@M<F=8J@KL8K@FEK?8KNFLC;D8B<=8CJ<+<I?8GJ=FIK?8KI<8JFEDP9<C@<=@E@JLE:FE;@K@FE8C

$ 8;D@K $ :8EEFK :FE:<@M< F= 8 `J@KL8K@FEa K?8K NFLC; make =8CJ< h/?<I< 8I< I<;<=@E@K@FEJ 9LK K?FJ< 8I< EFK `J@KL8eK@FEJa 8E; K?<E PFLcI< EF CFE><I K8CB@E> 89FLK FI i LKK?8K ;F<JEcK D8B< DP 9<C@<= LE:FE;@K@FE8C $ =@E; @K HL@K< <8JP KF@D8>@E<8J@KL8K@FEN?@:?NFLC; convince D<K?8K

.LGGFJ< $ >FK LG FE< DFIE@E> 8E; KFFB FLK KNF <8IGCL>J 8E;J<K K?<D ;FNE E<OK KF KNF FK?<I <8IGCL>J FE DP E@>?KK89C< 8E;EFK@:<; K?8K K?<I< N<I< EFN K?I<< <8IGCL>J N@K?FLK 8EP <8IGCL>J?8M@E> 8GG<8I<; FI ;@J8GG<8I<;g@E :FEKI8JK KF DP JKFI<; D<DFIP K?8K N8J JLGGFJ<; KF <HL8C (FI<FM<I N?<E $ M@JL8C@Q<;

K?< GIF:<JJ @E DP FNE D@E; @K J<<D<; K?8K D8B@E> 33 8E; 33:FD< FLK KF 3333 I<HL@I<; 8E <OKI8 3 KF 8GG<8I =IFD EFN?<I<8E; N8J DFI<FM<I @E:FEJ@JK<EK N@K? FK?<I 8I@K?D<K@: $ M@JL8C@Q<;J@E:< JL9KI8:K@E> 33 =IFD 333 C<=K 33 9LK JL9KI8:K@E> 33 =IFD3333 C<=K 333 /?@J NFLC; :FE=C@:K N@K? DP JKFI<; D<DFIP K?8K e 9LK D<DFIP NFLC; 9< 89JLI; @E K?< =8:< F= G?PJ@:8C8E;D<EK8C:FE=@ID8K@FEK?8K333e3333

http://lesswrong.com/lw/jr/how_to_convince_me_that_2_2_3/ 

+8><b2?8K@JM@;<E:<c http://lesswrong.com/lw/jl/what_is_evidence/f7h

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 93/309

$ NFLC; 8CJF :?<:B 8 GF:B<K :8C:LC8KFI "FF>C< 8E; G<I?8GJ DP :FGP F= N?<I< 2@EJKFE NI@K<J K?8K `!I<<;FD @J K?< =I<<;FDKF J8P KNF GCLJ KNF <HL8CJ K?I<<a CC F= K?<J< NFLC; E8KLI8CCP J?FN 

K?8K K?< I<JK F= K?< NFIC; 8>I<<; N@K? DP :LII<EK M@JL8C@Q8K@FE 8E;;@J8>I<<;N@K?DPD<DFIPK?8K

#FN :FLC; $ GFJJ@9CP ?8M< <M<I 9<<E JF ;<CL;<; 8J KF 9<C@<M<K?8K /NF <OGC8E8K@FEJ NFLC; :FD< KF D@E; !@IJK 8 E<LIFCF>@:8C =8LCK hGFJJ@9CP :8LJ<; 9P 8 JE<<Q<i ?8; D8;< 8CC K?<8;;@K@M< JLDJ @E DP JKFI<; D<DFIP >F LG 9P FE< .<:FE; JFD<eFE< N8J D<JJ@E> N@K? D< 9P ?PGEFJ@J FI 9P DP 9<@E> 8 :FDGLK<IJ@DLC8K@FE $E K?< J<:FE; :8J< $ NFLC; K?@EB @K DFI< C@B<CP K?8KK?<P ?8; D<JJ<; N@K? DP 8I@K?D<K@: recall  K?8E K?8K actually

<HL8CC<; )<@K?<I F= K?<J< GC8LJ@9C<eJFLE;@E> <OGC8E8K@FEJ NFLC;GI<M<EKD<=IFD EFK@:@E>K?8K$N8JM<IPM<IP very :FE=LJ<;

2?8K NFLC; :FEM@E:< D< K?8K @E FK?<I NFI;J @J <O8:KCP K?< J8D< B@E; F= <M@;<E:< K?8K :LII<EKCP :FEM@E:<J D< K?8K /?< <M@;<EK@8C :IFJJ=@I< F= G?PJ@:8C F9J<IM8K@FE D<EK8C M@JL8Ce@Q8K@FE8E;JF:@8C8>I<<D<EK

/?<I< N8J 8 K@D< N?<E $ ?8; EF @;<8 K?8K $ ;@; EFK 8IeI@M< 8K K?@J new 9<C@<= 9P I8E;FD GIF:<JJ<JgK?<E K?<I< NFLC; ?8M<

9<<E EF G8IK@:LC8I I<8JFE =FI DP 9I8@E KF <E; LG JKFI@E> ` a @EJK<8; F= ` a /?< =8:K K?8K DP 9I8@E JKFI<J 8E 8EJN<IJLIGI@J@E>CP J@D@C8I KF N?8K ?8GG<EJ N?<E $ C8P ;FNE KNF <8IGCL>J8CFE>J@;< KNF <8IGCL>J :8CCJ =FIK? 8E <OGC8E8K@FE F= N?8K <EK8E>C<eD<EKGIF;L:<JK?@JJKI8E><D@IIFI@E>F=D@E;8E;I<8C@KP

/?<I<cJ I<8CCP FECP KNF GFJJ@9@C@K@<J =FI 8 9<C@<= F= =8:K g<@K?<IK?< 9<C@<= >FK K?<I< M@8 8 D@E;eI<8C@KP <EK8E>C@E> GIF:<JJ FI EFK$= EFK K?< 9<C@<= :8EcK 9< :FII<:K <O:<GK 9P :F@E:@;<E:< !FI 9<C@<=J

 N@K? K?< JC@>?K<JK J?I<; F= @EK<IE8C :FDGC<O@KP  hI<HL@I@E> 8 :FDeGLK<I GIF>I8D F= DFI< K?8E 9@KJ KF J@DLC8K<i K?< JG8:< F= GFJJ@9@C@K@<J@JC8I><<EFL>?K?8K:F@E:@;<E:<M8E@J?<J

0E:FE;@K@FE8C =8:KJ 8I< EFK K?< J8D< 8J LE:FE;@K@FE8C 9<C@<=J$= <EK8E>C<; <M@;<E:< :FEM@E:<J D< K?8K 8 =8:K @J LE:FE;@K@FE8C K?@J;F<JEcK D<8E $ 8CN8PJ 9<C@<M<; @E K?< =8:K N@K?FLK E<<; F= <EK8Ee

 >C<;<M@;<E:<

+8><b4FLI.KI<E>K?8J8-8K@FE8C@JKc +8><b!<<C@E>-8K@FE8Cc

+8><b2?8K@JM@;<E:<c +8><b*::8DcJ-8QFIc

'/,''&.!&%,,

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 94/309

$ 9<C@<M< K?8K 8E; $ =@E; @K HL@K< <8JP KF :FE:<@M< F= 8 J@KL8K@FE N?@:? NFLC; :FEM@E:< D< K?8K )8D<CP K?< J8D<JFIK F= J@KL8K@FE K?8K :LII<EKCP :FEM@E:<J D< K?8K /?LJ $

;FEFK=<8IK?8K$8D8M@:K@DF=9C@E;=8@K?$= K?<I< 8I< 8EP ?I@JK@8EJ @E K?< 8L;@<E:< who know Bayes’s

Theorem hEF ELD<IFG?F9<J GC<8J<i D@>?K $ @EHL@I< F= PFL N?8K J@KeL8K@FE NFLC; :FEM@E:< PFL F= K?< KILK? F= $JC8D +I<JLD89CP @K

 NFLC; 9< K?< J8D< JFIK F= J@KL8K@FE :8LJ8CCP I<JGFEJ@9C< =FI GIFe;L:@E> PFLI :LII<EK 9<C@<= @E ?I@JK@8E@KP 2< NFLC; GLJ? PFLJ:I<8D@E> FLK F= K?< LK<ILJ F= 8 (LJC@D NFD8E 8E; ?8M< PFLI8@J<; 9P (LJC@D G8I<EKJ N?F :FEK@EL8CCP KFC; PFL K?8K @K @J >FF; KF9<C@<M< LE:FE;@K@FE8CCP @E $JC8D *I @J K?<I< DFI< KF @K K?8E K?8K

$= JF N?8K J@KL8K@FE NFLC; :FEM@E:< PFL F= $JC8D FI 8K C<8JK EFEe?I@JK@8E@KP

%(&,**!,'*1

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 95/309

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 96/309

:FDGC<O@KP F= 8E><I 8E; @E;<<; K?< :FDGC<O@KP F= @EK<CC@><E:< N8J >CFJJ<; FM<I 9P K?< ?LD8EJ N?F ?PGFK?<J@Q<; /?FI K?< K?LE;<Ie8><EK

To a human, (8ON<CCcJ HL8K@FEJ K8B< DL:? CFE><I KF <OGC8@EK?8E /?FI #LD8EJ ;FEcK ?8M< 8 9L@CKe@E MF:89LC8IP =FI :8C:LCLJK?< N8P N< ?8M< 8 9L@CKe@E MF:89LC8IP =FI 8E><I 4FLcM< >FK KF <OeGC8@E PFLI C8E>L8>< 8E; K?< C8E>L8>< 9<?@E; K?< C8E>L8>< 8E; K?<

 M<IP:FE:<GKF=D8K?<D8K@:J9<=FI<PFL:8EJK8IKFE<C<:KI@:@KP

 E; P<K @K J<<DJ K?8K K?<I< J?FLC; 9< JFD< J<EJ< @E N?@:?(8ON<CCcJ HL8K@FEJ 8I< simpler  K?8E 8 ?LD8E 9I8@E FI /?FI K?<K?LE;<Ie8><EK

/?<I< @J $KcJ enormously <8J@<I h8J @K KLIEJ FLKi KF NI@K< 8 :FDeGLK<I GIF>I8D K?8K J@DLC8K<J (8ON<CCcJ HL8K@FEJ :FDG8I<; KF8 :FDGLK<I GIF>I8D K?8K J@DLC8K<J 8E @EK<CC@><EK <DFK@FE8C D@E;C@B</?FI

/?< =FID8C@JD F= .FCFDFEF== $E;L:K@FE D<8JLI<J K?< `:FDeGC<O@KP F= 8 ;<J:I@GK@FEa 9P K?< C<E>K? F= K?< J?FIK<JK :FDGLK<IGIF>I8D N?@:? GIF;L:<J K?8K ;<J:I@GK@FE 8J 8E FLKGLK /F K8CB89FLK K?< `J?FIK<JK :FDGLK<I GIF>I8Da K?8K ;F<J JFD<K?@E> PFLE<<; KF JG<:@=P 8 JG8:< F= :FDGLK<I GIF>I8DJ N?@:? I<HL@I<J 8 

C8E>L8>< 8E; @EK<IGI<K<I .FCFDFEF== $E;L:K@FE LJ<J /LI@E> D8e:?@E<J FI I8K?<I 9@KJKI@E>J K?8K JG<:@=P /LI@E> D8:?@E<J 2?8K @=  PFL ;FEcK C@B< /LI@E> D8:?@E<J /?<E K?<I<cJ FECP 8 :FEJK8EK :FDeGC<O@KP G<E8CKP KF ;<J@>E PFLI FNE 0E@M<IJ8C /LI@E> (8:?@E< K?8K@EK<IGI<KJ N?8K<M<I :F;< PFL >@M< @K @E N?8K<M<I GIF>I8DD@E> C8Ee

 >L8>< PFL C@B< @==<I<EK @E;L:K@M< =FID8C@JDJ 8I< G<E8C@Q<; 9P 8  NFIJKe:8J< :FEJK8EK =8:KFI I<C8K@M< KF <8:? FK?<I :FII<JGFE;@E> KFK?<J@Q<F=8LE@M<IJ8C@EK<IGI<K<I=FIK?8K=FID8C@JD

$E K?< 9<KK<I h$(#*i M<IJ@FEJ F= .FCFDFEF== $E;L:K@FE K?<

:FDGLK<I GIF>I8D ;F<J EFK GIF;L:< 8 ;<K<ID@E@JK@: GI<;@:K@FE9LK 8JJ@>EJ GIF989@C@K@<J KF JKI@E>J !FI <O8DGC< N< :FLC; NI@K<8 GIF>I8D KF <OGC8@E 8 =8@I :F@E 9P NI@K@E> 8 GIF>I8D K?8K 8JJ@>EJ<HL8C GIF989@C@K@<J KF 8CC 6) JKI@E>J F= C<E>K? ) /?@J @J.FCFDFEF== $E;L:K@FEcJ 8GGIF8:? KF fitting K?< F9J<IM<; ;8K8 /?<?@>?<I K?< GIF989@C@KP 8 GIF>I8D 8JJ@>EJ KF K?< F9J<IM<; ;8K8 K?<9<KK<I K?8K GIF>I8D fits K?< ;8K8 E; GIF989@C@K@<J DLJK JLD KF JF =FI 8 GIF>I8D KF 9<KK<I `=@Ka FE< GFJJ@9@C@KP @K DLJK JK<8C GIF98e9@C@KP D8JJ =IFD JFD< FK?<I GFJJ@9@C@KP N?@:? N@CC K?<E `=@Ka DL:?

%(&,**!,'*1

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 97/309

DFI< GFFICP /?<I< @J EF JLG<I=8@I :F@E K?8K 8JJ@>EJ n GIF98e9@C@KPKF?<8;J8E;nGIF989@C@KPKFK8@CJ

#FN ;F N< KI8;< F== K?< =@K KF K?< ;8K8 8>8@EJK K?< :FDGC<O@KP 

F= K?< GIF>I8D $= PFL @>EFI< :FDGC<O@KP G<E8CK@<J 8E; K?@EB only89FLK =@K K?<E PFL N@CC 8CN8PJ GI<=<I GIF>I8DJ K?8K :C8@D KF ;<K<IeD@E@JK@:8CCP GI<;@:K K?< ;8K8 8JJ@>E @K n GIF989@C@KP $= K?< :F@EJ?FNJ `#//##/a K?<E K?< GIF>I8D N?@:? :C8@DJ K?8K K?< :F@E

 N8J =@O<; KF J?FN `#//##/a =@KJ K?< F9J<IM<; ;8K8 K@D<J 9<KeK<I K?8E K?< GIF>I8D N?@:? :C8@DJ K?< :F@E @J =8@I FEM<IJ<CP @= 

 PFL @>EFI< =@K 8E; :FEJ@;<I only :FDGC<O@KP K?<E K?< `=8@I :F@Ea ?PeGFK?<J@J N@CC 8CN8PJ J<<D J@DGC<I K?8E 8EP FK?<I ?PGFK?<J@J M<E@= K?< :F@E KLIEJ LG `#/##/###/####/#####/_a $Ee

;<<; K?< =8@I :F@E is J@DGC<I 8E; @K =@KJ K?@J ;8K8 <O8:KCP 8J N<CC 8J @K=@KJ 8EP FK?<I JKI@E> F= :F@E=C@GJgEF DFI< EF C<JJg9LK N< J<<8EFK?<I ?PGFK?<J@J J<<D@E> EFK KFF :FDGC@:8K<; K?8K =@KJ K?< ;8K8 DL:?9<KK<I

$= PFL C<K 8 GIF>I8D JKFI< FE< DFI< 9@E8IP 9@K F= @E=FID8K@FE @K N@CC 9< 89C< KF :LK ;FNE 8 JG8:< F= GFJJ@9@C@K@<J 9P ?8C= 8E; ?<E:<8JJ@>E KN@:< 8J DL:? GIF989@C@KP KF 8CC K?< GF@EKJ @E K?< I<D8@Ee@E> JG8:< /?@J JL>><JKJ K?8K FE< 9@K F= GIF>I8D :FDGC<O@KP J?FLC;:FJK at least  8 `=8:KFI F= KNF >8@Ea @E K?< =@K $= PFL KIP KF ;<J@>E8 :FDGLK<I GIF>I8D K?8K <OGC@:@KCP JKFI<J 8E FLK:FD< C@B< `#/e/##/a K?< J@O 9@KJ K?8K PFL CFJ< @E :FDGC<O@KP DLJK ;<JKIFP 8CCGC8LJ@9@C@KP >8@E<; 9P 8 e=FC; @DGIFM<D<EK @E =@K *K?<IN@J< PFL

 N@CCJFFE<IFIC8K<I;<:@;<K?8K8CC=8@I:F@EJ8I<=@O<;

0EC<JJ PFLI GIF>I8D @J 9<@E> JD8IK 8E; compressing K?< ;8K8 @KJ?FLC; ;F EF >FF; ALJK KF DFM< FE< 9@K =IFD K?< ;8K8 @EKF K?< GIFe

 >I8D;<J:I@GK@FE

/?< N8P .FCFDFEF== @E;L:K@FE NFIBJ KF GI<;@:K J<HL<E:<J @J

K?8K PFL JLD LG FM<I 8CC 8CCFN<; :FDGLK<I GIF>I8DJg@= 8EP GIFe >I8D @J 8CCFN<; .FCFDFEF== @E;L:K@FE 9<:FD<J LE:FDeGLK89C<gN@K? <8:? GIF>I8D ?8M@E> 8 GI@FI GIF989@C@KP F= hi KFK?< GFN<I F= @KJ :F;< C<E>K? @E 9@KJ 8E; <8:? GIF>I8D @J =LIK?<I

 N<@>?K<; 9P @KJ =@K KF 8CC ;8K8 F9J<IM<; JF =8I /?@J >@M<J PFL 8  N<@>?K<;D@OKLI<F=<OG<IKJK?8K:8EGI<;@:K=LKLI<9@KJ

/?< (@E@DLD (<JJ8>< '<E>K? =FID8C@JD @J E<8ICP <HL@M8C<EKKF .FCFDFEF== @E;L:K@FE 4FL J<E; 8 JKI@E> ;<J:I@9@E> 8 :F;< 8E;K?<E PFL J<E; 8 JKI@E> ;<J:I@9@E> K?< ;8K8 @E K?8K :F;< 2?@:?<Me

'%6+* 2'* 

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 98/309

<I <OGC8E8K@FE C<8;J KF K?< J?FIK<JK total D<JJ8>< @J K?< 9<JK $= PFLK?@EB F= K?< J<K F= 8CCFN89C< :F;<J 8J 8 JG8:< F= :FDGLK<I GIFe

 >I8DJ 8E; K?< :F;< ;<J:I@GK@FE C8E>L8>< 8J 8 LE@M<IJ8C D8:?@E<

K?<E (@E@DLD (<JJ8>< '<E>K? @J E<8ICP <HL@M8C<EK KF .FCFDFEF== @E;L:K@FE h)<8ICP 9<:8LJ< @K :?FFJ<J K?< shortest GIF>I8D I8K?<IK?8EJLDD@E>LGFM<I8CCGIF>I8DJi

/?@J C<KJ LJ J<< :C<8ICP K?< GIF9C<D N@K? LJ@E> `/?< C8;P ;FNEK?< JKI<<K @J 8 N@K:? J?< ;@; @Ka KF <OGC8@E K?< G8KK<IE @E K?<J<HL<E:< `a $= PFLcI< J<E;@E> 8 D<JJ8>< KF 8 =I@<E; KIPe@E> KF ;<J:I@9< K?< J<HL<E:< PFL F9J<IM<; PFL NFLC; ?8M< KF J8P`/?< C8;P ;FNE K?< JKI<<K @J 8 N@K:? J?< D8;< K?< J<HL<E:< :FD<FLK a 4FLI 8::LJ8K@FE F= N@K:?:I8=K NFLC;EcK C<K PFL

 shorten K?< I<JK F= K?< D<JJ8>< PFL NFLC; JK@CC ?8M< KF ;<J:I@9< @E=LCC;<K8@CK?<;8K8N?@:??<IN@K:?<IP:8LJ<;

2@K:?:I8=K D8P =@K FLI F9J<IM8K@FEJ @E K?< J<EJ< F= HL8C@K8K@M<CP  permitting  K?<D 9LK K?@J @J 9<:8LJ< N@K:?:I8=K G<ID@KJ everything C@B< J8P@E> `+?CF>@JKFE` .F <M<E 8=K<I PFL J8P `N@K:?a PFL JK@CC?8M< KF ;<J:I@9< 8CC K?< F9J<IM<; ;8K8 @E =LCC ;<K8@C 4FL ?8M< EFKcompressed the total length of the message describing your observations 9P KI8EJD@KK@E> K?< D<JJ8>< 89FLK N@K:?:I8=K PFL ?8M< J@DGCP 8;;<;

8LJ<C<JJGIFCF>L<@E:I<8J@E>K?<KFK8CC<E>K?/?< I<8C JE<8B@E<JJ N8J :FE:<8C<; @E K?< NFI; `@Ka F= ` N@K:?;@;@Ka N@K:?;@; what? 

*= :FLIJ< K?8EBJ KF ?@E;J@>?K 9@8J 8E; 8E:?FI@E>  8E; =8B<<OGC8E8K@FEJ 8E; =8B< :8LJ8C@KP  8E; GFJ@K@M< 9@8J 8E; DFK@M8K<;:F>E@K@FE @K D8P J<<D 8CC KFF F9M@FLJ K?8K @= 8 NFD8E @J 8 N@K:?F= course J?< NFLC; D8B< K?< :F@E :FD< LG LK F= K?@J$?8M<8CI<8;PJGFB<E

+8><b!8B<8LJ8C@KPc +8><b#@E;J@>?K9@8Jc +8><bE:?FI@E>8E;;ALJKD<EKc +8><b!8B<OGC8E8K@FEJc +8><b!8B<8LJ8C@KPc

+8><b+FJ@K@M<@8J'FFB$EKFK?<8IBc +8><b&EFN@E>9FLK@8J<J8E#LIK+<FGC<c

%(&,**!,'*1

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 99/309

9. The Lens That Sees Its Flaws

Continuation of: 2?8K@JM@;<E:<

'@>?K C<8M<J K?< .LE 8E; JKI@B<J PFLI J?F<C8:<J 8E; 9FLE:<J F==JFD< G?FKFEJ <EK<I K?< GLG@CJ F= PFLI <P<J 8E; JKI@B< PFLI I<K@E8K?< <E<I>P F= K?< G?FKFEJ KI@>><IJ E<LI8C @DGLCJ<J K?< E<LI8C @DeGLCJ<J 8I< KI8EJD@KK<; KF K?< M@JL8CeGIF:<JJ@E> 8I<8J F= K?< 9I8@E8E; K?<I< K?< FGK@:8C @E=FID8K@FE @J GIF:<JJ<; 8E; I<:FEJKIL:K<;@EKF 8 DF;<C K?8K @J I<:F>E@Q<; 8J 8E LEK@<; J?F<C8:< 8E; JF PFL9<C@<M<K?8KPFLIJ?F<C8:<J8I<LEK@<;

#<I< @J K?< J<:I<K F=  deliberate rationality— K?@J N?FC< <EK8E>C<e

D<EK GIF:<JJ @J EFK D8>@: 8E; PFL :8E understand  @K 4FL :8Eunderstand ?FN PFL J<< PFLI J?F<C8:<J 4FL :8E think 89FLK N?@:?JFIK F= K?@EB@E> GIF:<JJ<J N@CC :I<8K< 9<C@<=J N?@:? D@IIFI I<8C@KP8E;N?@:?K?@EB@E>GIF:<JJ<JN@CCEFK

(@:< :8E J<< 9LK K?<P :8EcK LE;<IJK8E; J<<@E> You :8E LE;<IeJK8E; J<<@E> 8E; 9<:8LJ< F= K?8K PFL :8E ;F K?@E>J N?@:? D@:<:8EEFK ;F /8B< 8 DFD<EK KF D8IM<C 8K K?@J =FI @K @J @E;<<; D8Ie

 M<CFLJ

(@:< J<< 9LK K?<P ;FEcK BEFN K?<P ?8M< M@JL8C :FIK<O<J JF K?<P :8EcK :FII<:K =FI FGK@:8C @CCLJ@FEJ DFLJ< C@M<J @E 8 D<EK8C NFIC;K?8K @E:CL;<J :8KJ ?FC<J :?<<J< 8E; DFLJ<KI8GJg9LK EFK DFLJ<9I8@EJ /?<@I :8D<I8 ;F<J EFK K8B< G@:KLI<J F= @KJ FNE C<EJ LK

 N< 8J ?LD8EJ :8E CFFB 8K 8 J<<D@E>CP 9@Q8II< @D8>< 8E; I<8C@Q<K?8K G8IK F= N?8K N<cI< J<<@E> @J K?< C<EJ @KJ<C= 4FL ;FEcK 8CN8PJ?8M< KF 9<C@<M< PFLI FNE <P<J 9LK PFL ?8M< KF I<8C@Q< K?8K PFL have<P<JgPFL DLJK ?8M< ;@JK@E:K D<EK8C 9L:B<KJ =FI K?< D8G 8E; K?<K<II@KFIP =FI K?< J<EJ<J 8E; I<8C@KP '<JK PFL K?@EB K?@J 8 KI@M@8C 89@Ce

@KPI<D<D9<I?FNI8I<@K@J@EK?<8E@D8CB@E>;FD/?< N?FC< @;<8 F= .:@<E:< @J J@DGCP I<=C<:K@M< I<8JFE@E> 89FLK

8 DFI< I<C@89C< GIF:<JJ =FI D8B@E> K?< :FEK<EKJ F= PFLI D@E; D@IIFIK?< :FEK<EKJ F= K?< NFIC; $K @J K?< JFIK F= K?@E> D@:< NFLC; E<M<I@EM<EK +FE;<I@E> K?@J 9LJ@E<JJ F= `G<I=FID@E> I<GC@:89C< <OG<I@eD<EKJ KF =8CJ@=P K?<FI@<Ja N< :8E J<< why @K NFIBJ .:@<E:< @J EFK 8 

http://lesswrong.com/lw/jm/the_lens_that_sees_its_flaws/  +8><b2?8K@JM@;<E:<c +8><b(PJK<I@FLJEJN<IJKF(PJK<I@FLJ,L<JK@FEJc

+8><b`.:@<E:<a8JLI@[email protected]<Ic https://reader009.{domain}/reader009/html5/0403/5ac2856a07ead/5ac285974e655.png 

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 100/309

J<G8I8K< D8>@JK<I@LD =8I 8N8P =IFD I<8C C@=< 8E; K?< LE;<IJK8E;e@E> F= FI;@E8IP DFIK8CJ .:@<E:< @J EFK JFD<K?@E> K?8K FECP 8GGC@<JKF K?< @EJ@;< F= C89FI8KFI@<J .:@<E:< @KJ<C= @J 8E LE;<IJK8E;89C<

GIF:<JJe@EeK?<eNFIC;K?8K:FII<C8K<J9I8@EJN@K?I<8C@KP.:@<E:< makes sense, N?<E PFL K?@EB 89FLK @K LK D@:< :8EcK

K?@EB 89FLK K?@EB@E> N?@:? @J N?P K?<P ;FEcK ?8M< .:@<E:< *E<J?FLC; EFK FM<ICFFB K?< NFE;<I F= K?@JgFI K?< GFK<EK@8C GFN<I @K9<JKFNJFELJ8J@E;@M@;L8CJEFKALJKJ:@<EK@=@:JF:@<K@<J

 ;D@KK<;CP LE;<IJK8E;@E> K?< <E>@E< F= K?FL>?K D8P 9< a little more complicated K?8E LE;<IJK8E;@E> 8 JK<8D <E>@E<g9LK @K @J EFK 8  fundamentally ;@==<I<EKK8JB

*E:< LGFE 8 K@D< $ N<EK KF !)<KcJ G?@CFJFG?P KF 8JB `F PFL 9<C@<M< 8 EL:C<8I N8I N@CC F::LI @E K?< E<OK P<8IJ $= EF N?P EFKa *E< G<IJFE N?F 8EJN<I<; K?< HL<JK@FE J8@; ?< ;@;EcK <OG<:K8 EL:C<8I N8I =FI P<8IJ 9<:8LJ< `CC F= K?< GC8P<IJ @EMFCM<; @E;<:@J@FEJ I<>8I;@E> EL:C<8I N8I 8I< EFK @EK<I<JK<; I@>?K EFNa `LK

 N?P <OK<E; K?8K FLK =FI P<8IJa $ 8JB<; `+LI< ?FG<a N8J ?@JI<GCP

-<=C<:K@E> FE K?@J N?FC< K?FL>?K GIF:<JJ N< :8E J<< N?P K?<K?FL>?K F= EL:C<8I N8I D8B<J K?< G<IJFE LE?8GGP 8E; N< :8E

J<< ?FN ?@J 9I8@E K?<I<=FI< I<A<:KJ K?< 9<C@<= LK @= PFL @D8>@E<8 9@CC@FE NFIC;JgM<I<KK 9I8E:?<J FI /<>D8IB ;LGC@:8K<J gK?@JK?FL>?K GIF:<JJ N@CC EFK JPJK<D8K@:8CCP :FII<C8K< FGK@D@JKJ KF9I8E:?<J @E N?@:? EF EL:C<8I N8I F::LIJ h.FD< :C<M<I =<CCFN @J9FLE; KF J8P `? 9LK J@E:< $ ?8M< ?FG< $cCC NFIB 8 C@KKC< ?8I;<I8K DP AF9 GLDG LG K?< >CF98C <:FEFDP 8E; K?LJ ?<CG KF GI<M<EK:FLEKI@<J =IFD JC@;@E> @EKF K?< 8E>IP 8E; ?FG<C<JJ JK8K< N?<I< ELe:C<8I N8I @J 8 GFJJ@9@C@KP .F K?< KNF <M<EKJ 8I< I<C8K<; 8=K<I 8CCa

 K K?@J GF@EK N< ?8M< KF ;I8> @E 8P<JcJ /?<FI<D 8E; D<8JLI<K?< :?8I>< F= <EK8E>C<D<EK HL8EK@K8K@M<CP 4FLI FGK@D@JK@: E8KLI<:8EEFK ?8M< that  C8I>< 8E <==<:K FE K?< NFIC; @K :8EEFK F= @KJ<C=;<:I<8J< K?< GIF989@C@KP F= EL:C<8I N8I 9P n FI ?FN<M<I DL:?

 PFLI FGK@D@JK@: E8KLI< J?@=K<; PFLI 9<C@<=J .?@=K@E> PFLI 9<C@<=J 9P 

http://lesswrong.com/lw/i8/religions_claim_to_be_nondisprovable/  http://lesswrong.com/lw/gv/outside_the_laboratory/  http://arxiv.org/abs/astro-ph/0302131

+8><b2?8K@JM@;<E:<c +8><bE$EKL@K@M<OGC8E8K@FEF=8P<J/?<FI<Dc

%(&,**!,'*1

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 101/309

8 C8I>< 8DFLEK ;L< KF 8E <M<EK K?8K FECP :8II@<J 8 M<IP K@EP :?8I><F=<EK8E>C<D<EKN@CCJK@CCD<JJLGPFLID8GG@E>i

/F 8JB N?@:? 9<C@<=J D8B< PFL ?8GGP @J KF KLIE @EN8I; EFK FLKe

 N8I;g@K K<CCJ PFL JFD<K?@E> 89FLK PFLIJ<C= 9LK @K @J EFK <M@;<E:<<EK8E>C<; N@K? K?< <EM@IFED<EK $ ?8M< EFK?@E> 8EPK?@E> 8>8@EJK?8GG@E<JJ 9LK @K J?FLC; =FCCFN =IFD  PFLI G@:KLI< F= K?< NFIC;I8K?<IK?8EK8DG<I@E>N@K?K?<D<EK8CG8@EK9ILJ?<J

$= PFL :8E J<< K?@Jg@= PFL :8E J<< K?8K ?FG< @J J?@=K@E> PFLI first-order  K?FL>?KJ 9P KFF C8I>< 8 ;<>I<<g@= PFL :8E LE;<IJK8E; PFLID@E; 8J 8 D8GG@E>e<E>@E< N@K? =C8NJ @E @KgK?<E PFL :8E 8GGCP 8 I<e=C<:K@M< :FII<:K@FE /?< 9I8@E @J 8 =C8N<; C<EJ K?IFL>? N?@:? KF J<<I<8C@KP /?@J @J KIL< F= 9FK? DFLJ< 9I8@EJ 8E; ?LD8E 9I8@EJ LK 8 ?LD8E 9I8@E @J 8 =C8N<; C<EJ K?8K :8E LE;<IJK8E; @KJ FNE =C8NJg@KJJPJK<D8K@: <IIFIJ @KJ 9@8J<Jg8E; 8GGCP J<:FE;eFI;<I :FII<:K@FEJ KFK?<D /?@J in practice, D8B<J K?< =C8N<; C<EJ =8I DFI< GFN<I=LC)FKG<I=<:K9LK=8IDFI<GFN<I=LC

+8><b!<<C@E>-8K@FE8Cc

,$&+,,++!,+$/+

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 102/309

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 103/309

 A sequence on how to see through the disguises of answersor beliefs or statements, that don’t answer or say or mean

 anything.

Part II

Mysterious Answers to MysteriousQuestions

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 104/309

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 105/309

1. Making Beliefs Pay Rent (in AnticipatedExperiences)

/?LJ9<>@EJK?<8E:@<EKG8I89C< If a tree falls in a forest and no one hears it, does it make a sound? One

 says, “Yes it does, for it makes vibrations in the air.” Another says, “No it  does not, for there is no auditory processing in any brain.” 

.LGGFJ< K?8K 8=K<I K?< KI<< =8CCJ K?< KNF N8CB @EKF K?< =FI<JKKF><K?<I 2@CC FE< <OG<:K KF J<< K?< KI<< =8CC<E KF K?< I@>?K 8E;K?< FK?<I <OG<:K KF J<< K?< KI<< =8CC<E KF K?< C<=K .LGGFJ< K?8K 9<e=FI< K?< KI<< =8CCJ K?< KNF C<8M< 8 JFLE; I<:FI;<I E<OK KF K?< KI<<

2FLC; FE< GC8P@E> 98:B K?< I<:FI;<I <OG<:K KF ?<8I JFD<K?@E> ;@==<I<EK =IFD K?< FK?<I .LGGFJ< K?<P 8KK8:? 8E <C<:KIF<E:<G?8CFe

 >I8G? KF 8EP 9I8@E @E K?< NFIC; NFLC; FE< <OG<:K KF J<< 8 ;@==<I<EKKI8:< K?8E K?< FK?<I /?FL>? K?< KNF 8I>L< FE< J8P@E> `)Fa 8E;K?< FK?<I J8P@E> `4<Ja K?<P ;F EFK 8EK@:@G8K< 8EP ;@==<I<EK <OG<I@e<E:<J /?< KNF K?@EB K?<P ?8M< ;@==<I<EK DF;<CJ F= K?< NFIC; 9LKK?<P ?8M< EF ;@==<I<E:< N@K? I<JG<:K KF N?8K K?<P <OG<:K N@CC happento K?<D

$KcJ K<DGK@E> KF KIP KF <C@D@E8K< K?@J D@JK8B< :C8JJ 9P @EJ@JK@E> 

K?8K K?< FECP C<>@K@D8K< B@E; F= 9<C@<= @J 8E 8EK@:@G8K@FE F= J<EJFIP <OG<I@<E:< LK K?< NFIC; ;F<J @E =8:K :FEK8@E DL:? K?8K @J EFKJ<EJ<; ;@I<:KCP 2< ;FEcK J<< K?< 8KFDJ LE;<ICP@E> K?< 9I@:B 9LKK?< 8KFDJ 8I< @E =8:K K?<I< /?<I< @J 8 =CFFI 9<E<8K? PFLI =<<K 9LK

 PFL ;FEcK experience K?< =CFFI ;@I<:KCP PFL J<< K?< C@>?K reflected =IFDK?< =CFFI FI I8K?<I PFL J<< N?8K PFLI I<K@E8 8E; M@JL8C :FIK<O ?8M<GIF:<JJ<; F= K?8K C@>?K /F @E=<I K?< =CFFI =IFD J<<@E> K?< =CFFI @J KFJK<G 98:B @EKF K?< LEJ<<E :8LJ<J F= <OG<I@<E:< $K D8P J<<D C@B< 8 

 M<IPJ?FIK8E;;@I<:KJK<G9LK@K@JJK@CC8JK<G

4FL JK8E; FE KFG F= 8 K8CC 9L@C;@E> E<OK KF 8 >I8E;=8K?<I :CF:B N@K? 8E ?FLI D@ELK< 8E; K@:B@E> J<:FE; ?8E; $E PFLI ?8E; @J 8 9FNC@E> 98CC 8E; PFL ;IFG @K F== K?< IFF= *E N?@:? K@:B F= K?<:CF:B N@CC PFL ?<8I K?< :I8J? F= K?< 9FNC@E> 98CC ?@KK@E> K?< >IFLE;

/F 8EJN<I GI<:@J<CP PFL DLJK LJ< 9<C@<=J C@B< Earth’s gravity is 9.8 meters per second per second, 8E; This building is around 120 meters tall./?<J< 9<C@<=J 8I< EFK NFI;C<JJ 8EK@:@G8K@FEJ F= 8 J<EJFIP <OG<I@<E:<K?<P 8I< M<I98Ce@J? GIFGFJ@K@FE8C $K GIF989CP ;F<J EFK <O8>><I8K<

http://lesswrong.com/lw/i3/making_beliefs_pay_rent_in_anticipated_experiences/ 

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 106/309

DL:? KF ;<J:I@9< K?<J< KNF 9<C@<=J 8J J<EK<E:<J D8;< FLK F= NFI;JLK K?<J< KNF 9<C@<=J ?8M< 8E @E=<I<EK@8C consequence K?8K @J 8 ;@I<:KJ<EJFIP 8EK@:@G8K@FEg@= K?< :CF:BcJ J<:FE; ?8E; @J FE K?< ELD<Ie

8C N?<E PFL ;IFG K?< 98CC PFL 8EK@:@G8K< J<<@E> @K FE K?< ELD<I8C N?<E PFL ?<8I K?< :I8J? =@M< J<:FE;J C8K<I /F 8EK@:@G8K< J<EJFIP <OG<I@<E:<J 8J GI<:@J<CP 8J GFJJ@9C< N< DLJK GIF:<JJ 9<C@<=J K?8K 8I<EFK8EK@:@G8K@FEJF=J<EJFIP<OG<I@<E:<

$K @J 8 >I<8K JKI<E>K? F= Homo sapiens K?8K N< :8E 9<KK<I K?8E 8EP FK?<I JG<:@<J @E K?< NFIC; C<8IE KF DF;<C K?< LEJ<<E $K @J 8CJF FE<F= FLI >I<8K N<8B GF@EKJ #LD8EJ F=K<E 9<C@<M< @E K?@E>J K?8K 8I<EFKFECPLEJ<<E9LKLEI<8C

/?< J8D< 9I8@E K?8K 9L@C;J 8 E<KNFIB F= @E=<II<; :8LJ<J 9<?@E;

J<EJFIP <OG<I@<E:< :8E 8CJF 9L@C; 8 E<KNFIB F= :8LJ<J K?8K @J EFK:FEE<:K<; KF J<EJFIP <OG<I@<E:< FI GFFICP :FEE<:K<; C:?<D@JKJ9<C@<M<; K?8K G?CF>@JKFE :8LJ<; =@I<gN< :FLC; FM<IJ@DGCP K?<@ID@E;J 9P ;I8N@E> 8 C@KKC< EF;< C89<C<; `+?CF>@JKFEa 8E; 8E 8IIFN =IFD K?@J EF;< KF K?<@I J<EJFIP <OG<I@<E:< F= 8 :I8:BC@E> :8DGe=@I<g9LK K?@J 9<C@<= P@<C;<; EF 8;M8E:< GI<;@:K@FEJ K?< C@EB =IFDG?CF>@JKFE KF <OG<I@<E:< N8J 8CN8PJ :FE=@>LI<; 8=K<I K?< <OG<I@e<E:< I8K?<I K?8E :FEJKI8@E@E> K?< <OG<I@<E:< @E 8;M8E:< *I JLGeGFJ< PFLI GFJKDF;<IE E>C@J? GIF=<JJFI K<8:?<J PFL K?8K K?< =8eDFLJ NI@K<I 2LCBP 2@CB@EJ<E @J 8:KL8CCP 8 `GFJKeLKFG@8Ea 2?8K;F<J K?@J D<8E PFL J?FLC; <OG<:K =IFD ?@J 9FFBJ )FK?@E> /?<9<C@<= @= PFL :8E :8CC @K K?8K ;F<JEcK :FEE<:K KF J<EJFIP <OG<I@<E:<8K 8CC LK PFL ?8; 9<KK<I I<D<D9<I K?< GIFGFJ@K@FE8C 8JJ<IK@FE K?8K`2LCBP 2@CB@EJ<Ea ?8J K?< `GFJKeLKFG@8Ea 8KKI@9LK< JF PFL :8E I<e

 >LI>@K8K< @K FE K?< LG:FD@E> HL@Q '@B<N@J< @= `GFJKeLKFG@8EJa J?FN `:FCFE@8C 8C@<E8K@FEa @= K?< HL@Q 8JBJ N?<K?<I 2LCBP 2@CB@EJ<EJ?FNJ :FCFE@8C 8C@<E8K@FE PFLc; 9<KK<I 8EJN<I P<J /?< 9<C@<=J 8I<

:FEE<:K<; KF <8:? FK?<I K?FL>? JK@CC EFK :FEE<:K<; KF 8EP 8EK@:@eG8K<;<OG<I@<E:<

2< :8E 9L@C; LG N?FC< E<KNFIBJ F= 9<C@<=J K?8K 8I< :FEE<:K<;FECP KF <8:? FK?<Ig:8CC K?<J< `=CF8K@E>a 9<C@<=J $K @J 8 LE@HL<CP ?LeD8E =C8N 8DFE> 8E@D8C JG<:@<J 8 G<IM<IJ@FE F= Homo sapiens’s 89@C@KP KF9L@C;DFI<><E<I8C8E;=C<O@9C<9<C@<=E<KNFIBJ

/?< I8K@FE8C@JK M@IKL< F= empiricism :FEJ@JKJ F= :FEJK8EKCP 8JB@E>  N?@:? <OG<I@<E:<J FLI 9<C@<=J GI<;@:KgFI 9<KK<I P<K GIF?@9@K F PFL 9<C@<M< K?8K G?CF>@JKFE @J K?< :8LJ< F= =@I< /?<E N?8K ;F

 PFL <OG<:K KF J<< ?8GG<E 9<:8LJ< F= K?8K F PFL 9<C@<M< K?8K

%1+,*!'-+ &+/*+,'%1+,*!'-+)-+,!'&+

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 107/309

2LCBP 2@CB@EJ<E @J 8 GFJKeLKFG@8E /?<E N?8K ;F PFL <OG<:K KFJ<< 9<:8LJ< F= K?8K )F EFK `:FCFE@8C 8C@<E8K@FEa what experiencewill happen to you? F PFL 9<C@<M< K?8K @= 8 KI<< =8CCJ @E K?< =FI<JK 8E;

EF FE< ?<8IJ @K @K JK@CC D8B<J 8 JFLE; /?<E N?8K <OG<I@<E:< DLJKK?<I<=FI<9<=8CCPFL

$K @J <M<E 9<KK<I KF 8JB N?8K <OG<I@<E:< must not  ?8GG<E KF PFL F PFL 9<C@<M< K?8K elan vital <OGC8@EJ K?< DPJK<I@FLJ 8C@M<E<JJF= C@M@E> 9<@E>J /?<E N?8K ;F<J K?@J 9<C@<=  not  8CCFN KF ?8GeG<EgN?8K NFLC; ;<=@E@K<CP =8CJ@=P K?@J 9<C@<= ELCC 8EJN<I D<8EJK?8K PFLI 9<C@<= ;F<J EFK constrain <OG<I@<E:< @K G<ID@KJ anything KF?8GG<EKFPFL $K=CF8KJ

2?<E PFL 8I>L< 8 J<<D@E>CP =8:KL8C HL<JK@FE 8CN8PJ B<<G @E

D@E; N?@:? ;@==<I<E:< F= 8EK@:@G8K@FE PFL 8I< 8I>L@E> 89FLK $= PFL:8EcK =@E; K?< ;@==<I<E:< F= 8EK@:@G8K@FE PFLcI< GIF989CP 8I>L@E> 89FLK C89<CJ @E PFLI 9<C@<= E<KNFIBgFI <M<E NFIJ< =CF8K@E> 9<C@<=J98IE8:C<J FE PFLI E<KNFIB $= PFL ;FEcK BEFN N?8K <OG<I@<E:<J 8I<@DGC@<; 9P 2LCBP 2@CB@EJ<E 9<@E> 8 GFJKeLKFG@8E PFL :8E >F FE8I>L@E>=FI<M<Ih4FL:8E8CJFGL9C@J?G8G<IJ=FI<M<Ii

 9FM< 8CC ;FEcK 8JB N?8K KF 9<C@<M<g8JB N?8K KF 8EK@:@G8K< Me<IP HL<JK@FE F= 9<C@<= J?FLC; =CFN =IFD 8 HL<JK@FE F= 8EK@:@G8K@FE

8E; K?8K HL<JK@FE F= 8EK@:@G8K@FE J?FLC; 9< K?< :<EK<I F= K?< @EeHL@IP M<IP >L<JJ F= 9<C@<= J?FLC; 9<>@E 9P =CFN@E> KF 8 JG<:@=@: >L<JJ F= 8EK@:@G8K@FE 8E; J?FLC; :FEK@EL< KF G8P I<EK @E =LKLI< 8EeK@:@G8K@FEJ$=89<C@<=KLIEJ;<8;9<8K<M@:K@K

%#!&$!+(1* &,!& &,!!(,0(*!&+

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 108/309

2. Belief in Belief 

Followup to: (8B@E> <C@<=J +8P -<EK h@E EK@:@G8K<; OG<I@e

<E:<Ji

8IC .8>8E FE:< KFC; 8 G8I89C< F= 8 D8E N?F :FD<J KF LJ 8E;:C8@DJ `/?<I< @J 8 ;I8>FE @E DP >8I8><a !8J:@E8K@E> 2< I<GCP K?8K

 N< N@J? KF J<< K?@J ;I8>FEgC<K LJ J<K FLK 8K FE:< =FI K?< >8I8><`LKN8@KaK?<:C8@D8EKJ8PJKFLJ`@K@J8E invisible ;I8>FEa

)FN 8J .8>8E GF@EKJ FLK K?@J ;F<JEcK D8B< K?< ?PGFK?<J@J LEe=8CJ@=@89C< +<I?8GJ N< >F KF K?< :C8@D8EKcJ >8I8>< 8E; 8CK?FL>? N<J<< EF ;I8>FE N< ?<8I ?<8MP 9I<8K?@E> =IFD EF M@J@9C< JFLI:< =FFKe

GI@EKJ DPJK<I@FLJCP 8GG<8I FE K?< >IFLE; 8E; @EJKILD<EKJ J?FN K?8K JFD<K?@E> @E K?< >8I8>< @J :FEJLD@E> FOP><E 8E; 9I<8K?@E> FLK:8I9FE;@FO@;<

LK EFN JLGGFJ< K?8K N< J8P KF K?< :C8@D8EK `*B8P N<cCC M@J@KK?< >8I8>< 8E; J<< @= N< :8E ?<8I ?<8MP 9I<8K?@E>a 8E; K?< :C8@D8EKHL@:BCP J8PJ EF @KcJ 8E inaudible ;I8>FE 2< GIFGFJ< KF D<8JLI< :8Ie9FE ;@FO@;< @E K?< 8@I 8E; K?< :C8@D8EK J8PJ K?< ;I8>FE ;F<J EFK9I<8K?< 2< GIFGFJ< KF KFJJ 8 98> F= =CFLI @EKF K?< 8@I KF J<< @= @K FLKC@E<J 8E @EM@J@9C< ;I8>FE 8E; K?< :C8@D8EK @DD<;@8K<CP J8PJ

`/?<;I8>FE@JG<ID<89C<KF=CFLIa8IC .8>8E LJ<; K?@J G8I89C< KF @CCLJKI8K< K?< :C8JJ@: DFI8C K?8K

GFFI ?PGFK?<J<J E<<; KF ;F =8JK =FFKNFIB KF 8MF@; =8CJ@=@:8K@FE LK$ K<CC K?@J G8I89C< KF D8B< 8 ;@==<I<EK GF@EK /?< :C8@D8EK DLJK ?8M<8E 8::LI8K< DF;<C F= K?< J@KL8K@FE somewhere @E ?@J D@E; 9<:8LJ<?< :8E 8EK@:@G8K< @E 8;M8E:< exactly which experimental results he’ll  need to excuse.

.FD< G?@CFJFG?<IJ ?8M< 9<<E DL:? :FE=LJ<; 9P JL:? J:<E8I@FJ

8JB@E> `F<J K?< :C8@D8EK really 9<C@<M< K?<I<cJ 8 ;I8>FE GI<J<EKFI EFKa J @= K?< ?LD8E 9I8@E FECP ?8; <EFL>? ;@JB JG8:< KF I<GI<eJ<EK FE< 9<C@<= 8K 8 K@D< -<8C D@E;J 8I< DFI< K8E>C<; K?8E K?8K J;@J:LJJ<; @E P<JK<I;8PcJ GFJK K?<I< 8I< ;@==<I<EK KPG<J F= 9<C@<= EFK8CC 9<C@<=J 8I< ;@I<:K 8EK@:@G8K@FEJ /?< :C8@D8EK :C<8ICP ;F<J EFK anticipate J<<@E> 8EPK?@E> LELJL8C LGFE FG<E@E> K?< >8I8>< ;FFIFK?<IN@J< ?< NFLC;EcK D8B< 8;M8E:< <O:LJ<J $K D8P 8CJF 9< K?8K

http://lesswrong.com/lw/i4/belief_in_belief/  +8><b(8B@E><C@<=J+8P-<EKh@EEK@:@G8K<;OG<I@<E:<Jic

http://www.godlessgeeks.com/LINKS/Dragon.htm +8><b(8B@E><C@<=J+8P-<EKh@EEK@:@G8K<;OG<I@<E:<Jic

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 109/309

K?< :C8@D8EKcJ GFFC F= GIFGFJ@K@FE8C 9<C@<=J :FEK8@EJ There is a drag-on in my garage. $K D8P J<<D KF 8 I8K@FE8C@JK K?8K K?<J< KNF 9<C@<=JJ?FLC; :FCC@;< 8E; :FE=C@:K <M<E K?FL>? K?<P 8I< F= ;@==<I<EK KPG<J

4<K @K @J 8 G?PJ@:8C =8:K K?8K PFL :8E NI@K< `/?< JBP @J >I<<Ea E<OKKF8G@:KLI<F=89CL<JBPN@K?FLKK?<G8G<I9LIJK@E>@EKF=C8D<J

/?< I8K@FE8C@JK M@IKL< F= <DG@I@:@JD @J JLGGFJ<; KF GI<M<EK LJ=IFD K?@J :C8JJ F= D@JK8B< 2<cI< JLGGFJ<; KF :FEJK8EKCP 8JB FLI9<C@<=J N?@:? <OG<I@<E:<J K?<P GI<;@:K D8B< K?<D G8P I<EK @E8EK@:@G8K@FE LK K?< ;I8>FEe:C8@D8EKcJ GIF9C<D ILEJ ;<<G<I 8E;:8EEFK 9< :LI<; N@K? JL:? J@DGC< 8;M@:< $KcJ EFK <O8:KCP difficult KF:FEE<:K 9<C@<= @E 8 ;I8>FE KF 8EK@:@G8K<; <OG<I@<E:< F= K?< >8I8><$= PFL 9<C@<M< K?<I<cJ 8 ;I8>FE @E PFLI >8I8>< K?<E PFL :8E <OG<:K

KF FG<E LG K?< ;FFI 8E; J<< 8 ;I8>FE $= PFL ;FEcK J<< 8 ;I8>FEK?<E K?8K D<8EJ K?<I<cJ EF ;I8>FE @E PFLI >8I8>< /?@J @J GI<KKP JKI8@>?K=FIN8I;4FL:8E<M<EKIP@KN@K?PFLIFNE>8I8><

)F K?@J @EM@J@9@C@KP 9LJ@E<JJ @J 8 JPDGKFD F= JFD<K?@E> DL:? NFIJ<

<G<E;@E> FE ?FN PFLI :?@C;?FF; N<EK PFL D8P I<D<D9<I 8 K@D< G<I@F; N?<E PFL =@IJK 9<>8E KF ;FL9K .8EK8 C8LJcJ <O@JK<E:<9LK PFL JK@CC 9<C@<M<; K?8K PFL N<I< supposed  KF 9<C@<M< @E .8EK8 

C8LJ JF PFL KI@<; KF ;<EP K?< ;FL9KJ J 8E@<C <EE<KK F9J<IM<J N?<I< @K @J ;@==@:LCK KF 9<C@<M< 8 K?@E> @K @J F=K<E DL:? <8J@<I KF9<C@<M< K?8K PFL ought  KF 9<C@<M< @K 2?8K ;F<J @K D<8E KF 9<C@<M<K?8K K?< 0CK@D8K< FJD@: .BP  @J 9FK? G<I=<:KCP 9CL< 8E; G<I=<:KCP 

 >I<<E /?< JK8K<D<EK @J :FE=LJ@E> @KcJ EFK <M<E :C<8I N?8K @K NFLC; mean KF 9<C@<M< @KgN?8K <O8:KCP NFLC; be 9<C@<M<; @= PFL 9<C@<M<;4FL :8E DL:? DFI< <8J@CP 9<C@<M< K?8K @K @J proper, K?8K @K @J good 8E;virtuous 8E; beneficial, KF 9<C@<M< K?8K K?< 0CK@D8K< FJD@: .BP @J9FK? G<I=<:KCP 9CL< 8E; G<I=<:KCP >I<<E <EE<KK :8CCJ K?@J `9<C@<= 

@E9<C@<=a E; ?<I< K?@E>J 9<:FD< :FDGC@:8K<; 8J ?LD8E D@E;J 8I< NFEK

KF ;Fg$ K?@EB <M<E <EE<KK FM<IJ@DGC@=@<J ?FN K?@J GJP:?FCF>P  NFIBJ @E GI8:K@:< !FI FE< K?@E> @= PFL 9<C@<M< @E 9<C@<= PFL :8EEFK8;D@K KF PFLIJ<C= K?8K PFL FECP 9<C@<M< @E 9<C@<= 9<:8LJ< @K @J M@IKLeFLJ KF believe, EFK KF 9<C@<M< @E 9<C@<= 8E; JF @= PFL FECP 9<C@<M< @E9<C@<= @EJK<8; F= 9<C@<M@E> PFL 8I< EFK M@IKLFLJ )F9F;P N@CC admit KF K?<DJ<CM<J `$ ;FEcK 9<C@<M< K?< 0CK@D8K< FJD@: .BP @J 9CL< 8E;

+8><b!89C<F=.:@<E:<8E;+FC@K@:Jc

$!!&$!

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 110/309

 >I<<E 9LK $ 9<C@<M< $ FL>?K KF 9<C@<M< @KagEFK LEC<JJ K?<P 8I< LEeLJL8CCP :8G89C< F= 8:BEFNC<;>@E> K?<@I FNE C8:B F= M@IKL< +<FGC<;FEcK9<C@<M<@E9<C@<=@E9<C@<=K?<PALJK9<C@<M<@E9<C@<=

h/?FJ< N?F =@E; K?@J :FE=LJ@E> D8P =@E; @K ?<CG=LC KF JKL;P D8K?<D8K@:8C CF>@: N?@:? KI8@EJ FE< KF D8B< M<IP J?8IG ;@JK@E:eK@FEJ 9<KN<<E K?< GIFGFJ@K@FE + 8 GIFF= F= + 8E; 8 GIFF= K?8K + @JGIFM89C< /?<I< 8I< J@D@C8ICP J?8IG ;@JK@E:K@FEJ 9<KN<<E + N8EK@E> + 9<C@<M@E> + N8EK@E> KF 9<C@<M< + 8E; 9<C@<M@E> K?8K PFL 9<C@<M<+i

/?<I<cJ ;@==<I<EK B@E;J F= 9<C@<= @E 9<C@<= 4FL D8P 9<C@<M< @E9<C@<= <OGC@:@KCP PFL D8P I<:@K< @E PFLI ;<C@9<I8K< JKI<8D F= :FEeJ:@FLJE<JJ K?< M<I98C J<EK<E:< `$K @J M@IKLFLJ KF 9<C@<M< K?8K K?<

0CK@D8K< FJD@: .BP @J G<I=<:KCP 9CL< 8E; G<I=<:KCP >I<<Ea h2?@C<8CJF 9<C@<M@E> K?8K PFL 9<C@<M< K?@J LEC<JJ PFL 8I< LELJL8CCP :8eG89C< F= 8:BEFNC<;>@E> PFLI FNE C8:B F= M@IKL<i LK K?<I<cJ 8CJFC<JJ <OGC@:@K =FIDJ F= 9<C@<= @E 9<C@<= (8P9< K?< ;I8>FEe:C8@D8EK=<8IJ K?< GL9C@: I@;@:LC< K?8K ?< @D8>@E<J N@CC I<JLCK @= ?< GL9C@:CP :FE=<JJ<J ?< N8J NIFE> h8CK?FL>? @E =8:K 8 I8K@FE8C@JK NFLC; :FEe

 >I8KLC8K< ?@D 8E; FK?<IJ 8I< DFI< C@B<CP KF I@;@:LC< ?@D @= ?< >F<JFE :C8@D@E> K?<I<cJ 8 ;I8>FE @E ?@J >8I8><i (8P9< K?< ;I8>FEe:C8@D8EK =C@E:?<J 8N8P =IFD K?< GIFJG<:K F= 8;D@KK@E> KF ?@DJ<C= K?8K K?<I< @J EF ;I8>FE 9<:8LJ< @K :FE=C@:KJ N@K? ?@J J<C=e@D8>< 8JK?< >CFI@FLJ ;@J:FM<I<I F= K?< ;I8>FE N?F J8N @E ?@J >8I8>< N?8K 8CCFK?<IJ?8;=8@C<;KFJ<<

$= 8CC FLI K?FL>?KJ N<I< ;<C@9<I8K< M<I98C J<EK<E:<J C@B< G?@CFJFeG?<IJ D8E@GLC8K< K?< ?LD8E D@E; NFLC; 9< 8 >I<8K ;<8C <8J@<I =FI?LD8EJ KF LE;<IJK8E; !C<<K@E> D<EK8C @D8><J LEJGFB<E =C@E:?<J;<J@I<J 8:K<; LGFE N@K?FLK 8:BEFNC<;><D<EKgK?<J< 8::FLEK =FI8JDL:?F=FLIJ<CM<J8JNFI;J

2?@C< $ ;@J8>I<< N@K? <EE<KK FE JFD< ;<K8@CJ 8E; :FDGC@:8eK@FEJ $ JK@CC K?@EB K?8K <EE<KKcJ EFK@FE F= belief in belief  @J K?< B<P @EJ@>?K E<:<JJ8IP KF LE;<IJK8E; K?< ;I8>FEe:C8@D8EK LK N< E<<;8 N@;<I :FE:<GK F=  belief, EFK C@D@K<; KF M<I98C J<EK<E:<J `<C@<=aJ?FLC; @E:CL;< LEJGFB<E 8EK@:@G8K@FEe:FEKIFCC<IJ `<C@<= @E 9<C@<=aJ?FLC; @E:CL;< LEJGFB<E :F>E@K@M<e9<?8M@FIe>L@;<IJ $K @J EFK GJPe:?FCF>@:8CCP I<8C@JK@: KF J8P `/?< ;I8>FEe:C8@D8EK ;F<J EFK 9<C@<M<K?<I< @J 8 ;I8>FE @E ?@J >8I8>< ?< 9<C@<M<J @K @J 9<E<=@:@8C KF 9<C@<M<K?<I< @J 8 ;I8>FE @E ?@J >8I8><a LK @K @J I<8C@JK@: KF J8P K?< ;I8>FEe

%1+,*!'-+ &+/*+,'%1+,*!'-+)-+,!'&+

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 111/309

:C8@D8EK anticipates as if K?<I< @J EF ;I8>FE @E ?@J >8I8>< 8E; makesexcuses as if ?<9<C@<M<;@EK?<9<C@<=

4FL :8E GFJJ<JJ 8E FI;@E8IP D<EK8C G@:KLI< F= PFLI >8I8>< N@K?

EF ;I8>FEJ @E @K N?@:? :FII<:KCP GI<;@:KJ PFLI <OG<I@<E:<J FE FG<Ee@E> K?< ;FFI 8E; E<M<I FE:< K?@EB K?< M<I98C G?I8J< There is no dragon in my garage. $ <M<E 9<K @KcJ ?8GG<E<; KF PFLgK?8K N?<E PFLFG<E PFLI >8I8>< ;FFI FI 9<;IFFD ;FFI FI N?8K<M<I 8E; <OG<:K KFJ<<EF;I8>FEJEFJL:?M<I98CG?I8J<ILEJK?IFL>?PFLID@E;

 E; KF =C@E:? 8N8P =IFD >@M@E> LG PFLI 9<C@<= @E K?< ;I8>FEgFI=C@E:? 8N8P =IFD >@M@E> LG PFLI self-image 8J 8 G<IJFE N?F 9<C@<M<J@E K?< ;I8>FEg@K @J EFK E<:<JJ8IP KF <OGC@:@KCP K?@EB I want to believethere’s a dragon in my garage. $K @J FECP E<:<JJ8IP KF =C@E:? 8N8P =IFD

K?<GIFJG<:KF=8;D@KK@E>PFL;FEcK9<C@<M</F :FII<:KCP 8EK@:@G8K< @E 8;M8E:< N?@:? <OG<I@D<EK8C I<JLCKJ

J?8CC E<<; KF 9< <O:LJ<; K?< ;I8>FEe:C8@D8EK DLJK h8i GFJJ<JJ 8E8::LI8K< 8EK@:@G8K@FEe:FEKIFCC@E> DF;<C JFD<N?<I< @E ?@J D@E;8E; h9i 8:K :F>E@K@M<CP KF GIFK<:K <@K?<I h9i ?@J =I<<e=CF8K@E> GIFGFeJ@K@FE8C 9<C@<= @E K?< ;I8>FE FI h9i ?@J J<C=e@D8>< F= 9<C@<M@E> @E K?<;I8>FE

$= JFD<FE< 9<C@<M<J @E K?<@I 9<C@<= @E K?< ;I8>FE 8E; 8CJF 9<e

C@<M<J @E K?< ;I8>FE K?< GIF9C<D @J DL:? C<JJ J<M<I< /?<P N@CC 9< N@CC@E> KF JK@:B K?<@I E<:B FLK FE <OG<I@D<EK8C GI<;@:K@FEJ 8E; G<Ie?8GJ <M<E 8>I<< KF >@M< LG K?< 9<C@<= @= K?< <OG<I@D<EK8C GI<;@:K@FE@J NIFE>g8CK?FL>? 9<C@<= @E 9<C@<= :8E JK@CC @EK<I=<I< N@K? K?@J @= K?<9<C@<= @KJ<C= @J EFK 89JFCLK<CP :FE=@;<EK 2?<E JFD<FE< D8B<J LG<O:LJ<J in advance, @K NFLC; J<<D KF I<HL@I< K?8K 9<C@<= 8E; 9<C@<= @E9<C@<=?8M<9<:FD<LEJPE:?IFE@Q<;

$!!&$!

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 112/309

3. Bayesian Judo

4FL :8E ?8M< JFD< =LE N@K? G<FGC< N?FJ< 8EK@:@G8K@FEJ ><K FLK F= 

JPE:N@K?N?8KK?<P9<C@<M<K?<P9<C@<M<$ N8J FE:< 8K 8 ;@EE<I G8IKP KIP@E> KF <OGC8@E KF 8 D8E N?8K $

;@; =FI 8 C@M@E> N?<E ?< J8@; `$ ;FEcK 9<C@<M< IK@=@:@8C $EK<CC@><E:<@JGFJJ@9C<9<:8LJ<FECP"F;:8ED8B<8JFLCa

 K K?@J GF@EK $ DLJK ?8M< 9<<E ;@M@E<CP @EJG@I<; 9<:8LJ< $@EJK8EKCP I<JGFE;<; `4FL D<8E @= $ :8E D8B< 8E IK@=@:@8C $EK<CC@e

 ><E:<@KGIFM<JPFLII<C@>@FE@J=8CJ<a

#<J8@;`2?8Ka

$ J8@; `2<CC @= PFLI I<C@>@FE GI<;@:KJ K?8K $ :8EcK GFJJ@9CP D8B<8E IK@=@:@8C $EK<CC@><E:< K?<E @= $ D8B< 8E IK@=@:@8C $EK<CC@><E:<@K D<8EJ PFLI I<C@>@FE @J =8CJ< @K?<I PFLI I<C@>@FE 8CCFNJ K?8K @KD@>?K 9< GFJJ@9C< =FI D< KF 9L@C; 8E $ FI @= $ 9L@C; 8E $ K?8K;@JGIFM<JPFLII<C@>@FEa

/?<I< N8J 8 G8LJ< 8J K?< FE< I<8C@Q<; ?< ?8; ALJK D8;< ?@J ?PeGFK?<J@J MLCE<I89C< KF =8CJ@=@:8K@FE 8E; K?<E ?< J8@; `2<CC $ ;@;EcKD<8E K?8K PFL :FLC;EcK D8B< 8E @EK<CC@><E:< ALJK K?8K @K :FLC;EcK 9<<DFK@FE8C@EK?<J8D<N8PN<8I<a

$ J8@; `.F @= $ D8B< 8E IK@=@:@8C $EK<CC@><E:< K?8K N@K?FLK9<@E> ;<C@9<I8K<CP GI<GIF>I8DD<; N@K? 8EP JFIK F= J:I@GK JK8IKJK8CB@E> 89FLK 8E <DFK@FE8C C@=< K?8K JFLE;J C@B< FLIJ that  D<8EJ

 PFLII<C@>@FE@JNIFE>a

#< J8@; `2<CC LD $ >L<JJ N< D8P ?8M< KF 8>I<< KF ;@J8>I<< FEK?@Ja

$ J8@; `)F N< :8EcK 8:KL8CCP /?<I<cJ 8 K?<FI<D F= I8K@FE8C@KP :8CC<; LD8EEcJ >I<<D<EK /?<FI<D N?@:? J?FNJ K?8K EF KNF I8e

K@FE8C@JKJ :8E 8>I<< KF ;@J8>I<< $= KNF G<FGC< ;@J8>I<< N@K? <8:?FK?<I8KC<8JKFE<F=K?<DDLJK9<;F@E>JFD<K?@E>NIFE>a

2< N<EK 98:B 8E; =FIK? FE K?@J 9I@<=CP !@E8CCP ?< J8@; `2<CC$ >L<JJ $ N8J I<8CCP KIP@E> KF J8P K?8K $ ;FEcK K?@EB PFL :8E D8B<JFD<K?@E><K<IE8Ca

$ J8@; `2<CC $ ;FEcK K?@EB JF <@K?<I $cD >C8; N< N<I< 89C<KF I<8:? 8>I<<D<EK FE K?@J 8J LD8EEcJ >I<<D<EK /?<FI<D I<e

http://lesswrong.com/lw/i5/bayesian_judo/  +8><b<C@<=@E<C@<=c

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 113/309

HL@I<Ja $ JKI<K:?<; FLK DP ?8E; 8E; ?< J?FFB @K 8E; K?<E ?< N8E;<I<;8N8P

  NFD8E N?F ?8; JKFF; E<8I9P C@JK<E@E> KF K?< :FEM<IJ8K@FE

J8@;KFD<>I8M<CP`/?8KN8J9<8LK@=LCa`/?8EBPFLM<IPDL:?a$J8@;

1+!&"-'

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 114/309

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 115/309

K?@J JKI8E>< 9<C@<=a 9LK `2?P ;F K?<P 8CC NI@K< K?@J JKI8E>< J<EeK<E:< FE HL@QQ<Ja M<E @= 8 J<EK<E:< @J <JJ<EK@8CCP D<8E@E>C<JJ PFL:8EJK@CCBEFNN?<EPFL8I<JLGGFJ<;KF:?8EKK?<I<JGFEJ<8CFL;

$ K?@EB <EE<KK D8P 9< JC@>?KCP KFF :PE@:8C @E JL>><JK@E> K?8KI<C@>@FLJ GIF=<JJ@FE @J just J8P@E> K?< 9<C@<= 8CFL;gDFJK G<FGC< 8I<?FE<JK <EFL>? K?8K @= K?<P J8P 8 I<C@>@FLJ JK8K<D<EK 8CFL; K?<P N@CC8CJF =<<C F9C@>8K<; KF J8P K?< M<I98C J<EK<E:< @EKF K?<@I FNE JKI<8DF=:FEJ:@FLJE<JJ

LK <M<E K?< :FE:<GK F= `I<C@>@FLJ GIF=<JJ@FEa ;F<JEcK J<<D KF:FM<I K?< G8>8E NFD8EcJ :C8@D KF 9<C@<M< @E K?< GI@DFI;@8C :FN$= PFL ?8; KF GIF=<JJ 8 I<C@>@FLJ 9<C@<= KF J8K@J=P 8 GI@<JK FI J8K@J=P 8 :FeI<C@>@FE@JKg?<:B KF J8K@J=P PFLI FNE J<C=e@D8>< 8J 8 I<C@>@FLJ

G<IJFEgPFL NFLC; ?8M< KF pretend KF 9<C@<M< much more convincinglyK?8E K?@J NFD8E N8J ;F@E> J J?< I<:@K<; ?<I K8C< F= K?< GI@DFIe;@8C :FN N@K? K?8K J8D< JKI8E>< =C8LEK@E> GI@;< J?< N8JEcK <M<Etrying KF 9< G<IJL8J@M<gN8JEcK <M<E KIP@E> KF :FEM@E:< LJ K?8K J?<KFFB ?<I FNE I<C@>@FE J<I@FLJCP $ K?@EB K?8KcJ K?< G8IK K?8K JF KFFBD< 898:B $ BEFN G<FGC< N?F 9<C@<M< K?<P 9<C@<M< I@;@:LCFLJ K?@E>J9LK N?<E K?<P GIF=<JJ K?<D K?<PcCC JG<E; DL:? DFI< <==FIK KF :FEe

 M@E:<K?<DJ<CM<JK?8KK?<PK8B<K?<@I9<C@<=JJ<I@FLJCP

$K =@E8CCP F::LII<; KF D< K?8K K?@J NFD8E N8JEcK KIP@E> KF :FEe M@E:< LJ FI <M<E :FEM@E:< ?<IJ<C= #<I I<:@K8K@FE F= K?< :I<8K@FEJKFIP N8JEcK about K?< :I<8K@FE F= K?< NFIC; 8K 8CC -8K?<I 9P C8LE:?e@E> @EKF 8 =@M<eD@ELK< ;@8KI@9< 89FLK K?< GI@DFI;@8C :FN J?< N8Jcheering for paganism, C@B< ?FC;@E> LG 8 98EE<I 8K 8 =FFK98CC >8D< 98EE<I J8P@E> `"* '0.a @JEcK 8 JK8K<D<EK F= =8:K FI 8E 8KK<DGKKFG<IJL8;<@K;F<JEcK?8M<KF9<:FEM@E:@E>g@KcJ8:?<<I

/?8K JKI8E>< =C8LEK@E> GI@;<_ @K N8J C@B< J?< N8J D8I:?@E> E8B<; @E 8 >8P GI@;< G8I8;< h$E:@;<EK8CCP $c; ?8M< EF F9A<:K@FE

@= J?< had  D8I:?<; E8B<; @E 8 >8P GI@;< G8I8;< '<J9@8E@JD @JEFK JFD<K?@E> K?8K KILK? :8E ;<JKIFP i $K N8JEcK ALJK 8 :?<<I C@B<D8I:?@E> 9LK 8E FLKI8><FLJ :?<<I C@B< D8I:?@E> E8B<;g9<C@<M@E> K?8K J?< :FLC;EcK 9< 8II<JK<; FI :I@K@:@Q<; 9<:8LJ< J?< N8J ;F@E> @K=FI?<IGI@;<G8I8;<

+8><b!89C<F=.:@<E:<8E;+FC@K@:Jc +8><b!<<C@E>-8K@FE8Cc

(*'++!&&*!&

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 116/309

/?8KcJ N?P @K D8KK<I<; KF ?<I K?8K N?8K J?< N8J J8P@E> N8J 9<e PFE; I@;@:LCFLJ $= J?<c; KI@<; KF D8B< @K JFLE; DFI< GC8LJ@9C< @K NFLC;?8M<9<<EC@B<GLKK@E>FE:CFK?<J

%1+,*!'-+ &+/*+,'%1+,*!'-+)-+,!'&+

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 117/309

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 118/309

=<JJ@E>  NFLC; ?8M< JFD< KIFL9C< :I<8K@E> ><EL@E< ;<<G GFN<I=LC<DFK@FE8C <==<:KJ *I JF $ JLJG<:K $ :FE=<JJ $cD EFK 8E <OG<IK?<I< LK DP @DGI<JJ@FE @J K?@J +<FGC< N?FcM< JKFGG<;

8EK@:@G8K@E>e8Je@= K?<@I I<C@>@FE @J KIL< N@CC >F KF >I<8K C<E>K?J KFconvince K?<DJ<CM<J K?<P 8I< G8JJ@FE8K< 8E; K?@J ;<JG<I8K@FE :8E 9<D@JK8B<E=FIG8JJ@FE LK@KcJEFKK?<J8D<=@I<K?<P?8;8J8:?@C;

*E K?< FK?<I ?8E; @K @J M<IP <8JP =FI 8 ?LD8E 9<@E> KF ><EeL@E<CP G8JJ@FE8K<CP >LKeC<M<C 9<CFE> KF 8 >IFLG KF :?<<I =FI K?<@I=8MFI@K< JGFIKJ K<8D h/?@J @J K?< =FLE;8K@FE FE N?@:? I<JKJ K?<JN@E;C< F= `-<GL9C@:8EJ MJ <DF:I8KJa 8E; 8E8CF>FLJ =8CJ< ;@C<DeD8J @E FK?<I :FLEKI@<J 9LK K?8KcJ 8 KFG@: =FI 8EFK?<I GFJKi $;<EeK@=P@E> N@K? 8 KI@9< @J 8 M<IP JKIFE> <DFK@FE8C =FI:< +<FGC< N@CC ;@<=FI @K E; FE:< PFL ><K G<FGC< KF @;<EK@=P N@K? 8 KI@9< K?< 9<C@<=J

 N?@:? 8I< 8KK@I< F= K?8K KI@9< N@CC 9< JGFB<E N@K? K?< =LCC G8JJ@FE F= 9<CFE>@E>KFK?8KKI@9<

+8><b+IF=<JJ@E>8E;?<<I@E>c

+8><b!89C<F=.:@<E:<8E;+FC@K@:Jc +8><b/?</?@I;CK<IE8K@M<c

%1+,*!'-+ &+/*+,'%1+,*!'-+)-+,!'&+

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 119/309

6. Focus Your Uncertainty 

2@CC 9FE; P@<C;J >F LG FI ;FNE FI I<D8@E K?< J8D< $= PFLcI< 8 /1

GLE;@K 8E; PFLI AF9 @J KF <OGC8@E K?< FLK:FD< 8=K<I K?< =8:K K?<EK?<I<cJ EF I<8JFE KF NFIIP )F D8KK<I which F= K?< K?I<< GFJJ@9@C@eK@<J :FD<J KIL< PFLcCC 9< 89C< KF <OGC8@E N?P K?< FLK:FD< G<I=<:KCP =@KJ PFLI G<K D8IB<K K?<FIP /?<I<cJ EF I<8JFE KF K?@EB F= K?<J<K?I<< GFJJ@9@C@K@<J 8J JFD<?FN opposed KF FE< 8EFK?<I 8J exclusive,9<:8LJ< PFLcCC ><K =LCC D8IBJ =FI GLE;@KIP EF D8KK<I N?@:? FLK:FD<F::LIJ

LK N8@K .LGGFJ< PFLcI< 8 novice /1 GLE;@K 8E; PFL 8I<EcK <OeG<I@<E:<; <EFL>? KF D8B< LG GC8LJ@9C< <OGC8E8K@FEJ FE K?< JGFK4FL E<<; KF GI<G8I< I<D8IBJ @E 8;M8E:< =FI KFDFIIFNcJ 9IF8;:8JK8E; PFL ?8M< C@D@K<; K@D< KF GI<G8I< $E K?@J :8J< @K NFLC; 9<?<CG=LC KF BEFN which FLK:FD< N@CC 8:KL8CCP F::LIgN?<K?<I 9FE;

 P@<C;J N@CC >F LG ;FNE FI I<D8@E K?< J8D<g9<:8LJ< K?<E PFL NFLC;FECPE<<;KFGI<G8I< one J<KF=<O:LJ<J

 C8J EF FE< :8E GFJJ@9CP =FI<J<< K?< =LKLI< 2?8K 8I< PFLKF ;F 4FL :<IK8@ECP :8EcK LJ< `GIF989@C@K@<Ja 2< 8CC BEFN =IFDJ:?FFC K?8K `GIF989@C@K@<Ja 8I< C@KKC< ELD9<IJ K?8K 8GG<8I E<OK KF

8 NFI; GIF9C<D 8E; K?<I< 8I<EcK 8EP C@KKC< ELD9<IJ ?<I< 2FIJ< PFL feel LE:<IK8@E 4FL ;FEcK I<D<D9<I feeling LE:<IK8@E N?@C< PFL N<I< D8E@GLC8K@E> K?< C@KKC< ELD9<IJ @E NFI; GIF9C<DJ College class-es teaching math 8I< E@:< :C<8E GC8:<J K?<I<=FI< math itself :8EcK 8GGCP KF C@=< J@KL8K@FEJ K?8K 8I<EcK E@:< 8E; :C<8E 4FL NFLC;EcK N8EK KF@E8GGIFGI@8K<CP KI8EJ=<I K?@EB@E> JB@CCJ =IFD FE< :FEK<OK KF 8EFK?e<I C<8ICPK?@J@JEFK8D8KK<I=FI`GIF989@C@K@<Ja

)FE<K?<C<JJ PFL FECP ?8M< D@ELK<J KF GI<G8I< PFLI <Oe:LJ<J 4FL :8EcK JG<E; K?< <EK@I< D@ELK<J FE `LGa 8E; 8CJFJG<E; 8CC D@ELK<J FE `;FNEa 8E; 8CJF JG<E; 8CC D@ELK<J FE`J8D<a4FLcM<>FKKFGI@FI@K@Q<JFD<?FN

$= PFL E<<;<; KF ALJK@=P PFLI K@D< <OG<E;@KLI< KF 8 I<M@<N :FDeD@KK<< PFL NFLC; ?8M< KF JG<E; <HL8C K@D< FE <8:? GFJJ@9@C@KP.@E:< K?<I< 8I< EF C@KKC< ELD9<IJ NI@KK<E ;FNE PFLc; ?8M< EF ;F:eLD<EK8K@FE KF ALJK@=P JG<E;@E> ;@==<I<EK 8DFLEKJ F= K@D< 4FL :8E?<8I K?< I<M@<N<IJ EFN And why, Mr. Finkledinger, did you spend ex-

http://lesswrong.com/lw/ia/focus_your_uncertainty/ 

http://lesswrong.com/lw/i2/two_more_things_to_unlearn_from_school/  http://www.aft.org/pubs-reports/american_educator/issues/summer07/Crit_Thinking.pdf 

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 120/309

 actly 42 minutes on excuse #3? Why not 41 minutes, or 43? Admit it—you’re not being objective! You’re playing subjective favorites! 

LK PFL I<8C@Q< N@K? 8 JD8CC =C8J? F= I<C@<= K?<I<cJ EF I<M@<N 

:FDD@KK<< KF J:FC; PFL /?@J @J >FF; 9<:8LJ< K?<I<cJ 8 D8AFI !<;e<I8C -<J<IM< 8EEFLE:<D<EK KFDFIIFN 8E; @K J<<DJ LEC@B<CP K?8K9FE; GI@:<J N@CC I<D8@E K?< J8D< 4FL ;FEcK N8EK KF JG<E; GI<e:@FLJD@ELK<JFE8E<O:LJ<PFL;FEcK8EK@:@G8K<E<<;@E>

4FLI D@E; B<<GJ ;I@=K@E> KF K?< <OGC8E8K@FEJ PFL LJ< FE K<C<e M@J@FE F= N?P <8:? <M<EK GC8LJ@9CP =@KJ PFLI D8IB<K K?<FIP LK @KI8G@;CP 9<:FD<J :C<8I K?8K GC8LJ@9@C@KP :8EcK ?<CG PFL ?<I<g8CC K?I<<<M<EKJ 8I< GC8LJ@9C< !@KK89@C@KP KF PFLI G<K D8IB<K K?<FIP ;F<JEcKK<CC PFL ?FN KF ;@M@;< PFLI K@D< /?<I<cJ 8E LE:IFJJ89C< >8G 9<e

KN<<E PFLI D@ELK<J F= K@D< N?@:? 8I< :FEJ<IM<; M<IJLJ PFLI89@C@KP KF <OGC8@E ?FN 8E FLK:FD< =@KJ PFLI K?<FIP N?@:? @J LEC@De@K<;

 E; P<K_ <M<E @E PFLI LE:<IK8@E JK8K< F= D@E; @K J<<DJ K?8K PFL anticipate K?< K?I<< <M<EKJ ;@==<I<EKCP K?8K PFL expect KF E<<; JFD<<O:LJ<J DFI< K?8E FK?<IJ E;gK?@J @J K?< =8J:@E8K@E> G8IKgN?<E

 PFL K?@EB F= JFD<K?@E> K?8K D8B<J @K J<<D more C@B<CP K?8K 9FE;GI@:<J N@CC >F LG K?<E PFL =<<C less C@B<CP KF E<<; 8E <O:LJ< =FI 9FE;

GI@:<J>F@E>;FNEFII<D8@E@E>K?<J8D<$K <M<E J<<DJ C@B< K?<I<cJ 8 I<C8K@FE 9<KN<<E ?FN DL:? PFL 8EeK@:@G8K< <8:? F= K?< K?I<< FLK:FD<J 8E; ?FN DL:? K@D< PFL N8EKKF JG<E; GI<G8I@E> <8:? <O:LJ< *= :FLIJ< K?< I<C8K@FE :8EcK 8:KL8CeCP 9< HL8EK@=@<; 4FL ?8M< D@ELK<J KF GI<G8I< PFLI JG<<:? 9LKK?<I< @JEcK F= 8EPK?@E> KF ;@M@;< LG @E K?@J 8EK@:@G8K@FE 9LJ@E<JJhCK?FL>? PFL ;F NFIB FLK K?8K if JFD< G8IK@:LC8I FLK:FD< F::LIJK?<E PFLI LK@C@KP =LE:K@FE @J CF>8I@K?D@: @E K@D< JG<EK GI<G8I@E> K?<<O:LJ<i

.K@CC_ PFLI D@E; B<<GJ :FD@E> 98:B KF K?< @;<8 K?8K 8EK@:@G8eK@FE @J C@D@K<; LEC@B< <O:LJ89@C@KP 9LK C@B< K@D< KF GI<G8I< <O:LJ<J(8P9< 8EK@:@G8K@FE J?FLC; 9< KI<8K<; 8J 8  conserved resource, C@B<DFE<P 4FLI =@IJK @DGLCJ< @J KF KIP KF ><K DFI< 8EK@:@G8K@FE 9LK

 PFL JFFE I<8C@Q< K?8K <M<E @= PFL ><K DFI< 8EK@:@GK8@FE PFL NFEcK?8M< 8EP DFI< K@D< KF GI<G8I< PFLI <O:LJ<J )F PFLI FECP :FLIJ<@JKF allocate PFLI limited supply F=8EK@:@G8K@FE8J9<JKPFL:8E

4FLcI< GI<KKP JLI< PFL N<I<EcK K8L>?K 8EPK?@E> C@B< K?8K @E PFLI

JK8K@JK@:J :FLIJ<J /?<P ;@;EcK K<CC PFL N?8K KF ;F N?<E PFL felt JF

%1+,*!'-+ &+/*+,'%1+,*!'-+)-+,!'&+

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 121/309

K<II@9CP LE:<IK8@E /?<P ;@;EcK K<CC PFL N?8K KF ;F N?<E K?<I< N<I<EF C@KKC< ELD9<IJ ?8E;<; KF PFL 2?P <M<E @= PFL KI@<; KF LJ<ELD9<IJ PFL D@>?K <E; LG LJ@E> 8EP JFIK F= ELD9<IJ 8K 8CCgK?<I<cJ

EF ?@EK N?8K B@E; F= D8K? KF LJ< @= PFL J?FLC; 9< LJ@E> D8K?(8P9< PFLc; <E; LG LJ@E> pairs F= ELD9<IJ I@>?K 8E; C<=K ELD9<IJ N?@:? PFLc; :8CC . =FI <OK<Ie.@E@JK<I_ FI N?F BEFNJ N?8K <CJ<h/?FL>? PFL ;F ?8M< FECP D@ELK<J KF JG<E; GI<G8I@E> <O:LJ<Ji

$= FECP K?<I< N<I< 8E 8IK F= focusing your uncertainty gF= squeezing 8J DL:? 8EK@:@G8K@FE 8J GFJJ@9C< @EKF N?@:?<M<I FLK:FD< N@CC actu- ally happen! 

LK N?8K :FLC; N< :8CC 8E 8IK C@B< K?8K E; N?8K NFLC; K?<ILC<J9<C@B<

'-+1'-* - &*,!&,1

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 122/309

7. The Virtue of Narrowness

What is true of one apple may not be true of another apple; thus more can be

 said about a single apple than about all the apples in the world. g/N<CM<1@IKL<JF=-8K@FE8C@KP 

2@K?@E K?<@I FNE GIF=<JJ@FEJ G<FGC< >I8JG K?< @DGFIK8E:< F= E8IIFNE<JJ 8 :8I D<:?8E@: BEFNJ K?< ;@==<I<E:< 9<KN<<E 8 :8Ie9LI<KFI 8E; 8 I8;@8KFI 8E; NFLC; EFK K?@EB F= K?<D 9FK? 8J `:8IG8IKJa ?LEK<Ie>8K?<I<I BEFNJ K?< ;@==<I<E:< 9<KN<<E 8 C@FE 8E;8 G8EK?<I A8E@KFI ;F<J EFK N@G< K?< =CFFI N@K? N@E;FN :C<8E<I<M<E @= K?< 9FKKC<J CFFB J@D@C8I KF FE< N?F ?8J EFK D8JK<I<; K?< 8IK

*LKJ@;< K?<@I FNE GIF=<JJ@FEJ G<FGC< F=K<E :FDD@K K?< D@JeJK<G F= KIP@E> KF 9IF8;<E 8 NFI; 8J N@;<CP 8J GFJJ@9C< KF :FM<I8J DL:? K<II@KFIP 8J GFJJ@9C< $J @K EFK DFI< >CFI@FLJ DFI< N@J<DFI< @DGI<JJ@M< KF K8CB 89FLK all  K?< 8GGC<J @E K?< NFIC; #FN DL:? CF=K@<I @K DLJK 9< KF explain human thought in general, N@K?FLK9<@E> ;@JKI8:K<; 9P JD8CC<I HL<JK@FEJ JL:? 8J ?FN ?LD8EJ @EM<EKK<:?E@HL<J =FI JFCM@E> 8 -L9@BcJ L9< $E;<<; @K J:8I:<CP J<<DJE<:<JJ8IP KF :FEJ@;<I specific HL<JK@FEJ 8K 8CC @JEcK 8 ><E<I8C K?<FIP 8 

 NFIK?P<EFL>?8::FDGC@J?D<EKFE@KJFNE

$K @J K?< N8P F= K?< :LI@FLJ KF C@=K LG FE< G<99C< =IFD 8DFE> 8 D@CC@FE G<99C<J FE K?< J?FI< 8E; J<< JFD<K?@E> E<N 89FLK @KJFD<K?@E> @EK<I<JK@E> JFD<K?@E> ;@==<I<EK 4FL :8CC K?<J< G<99C<J`;@8DFE;Ja 8E; 8JB N?8K D@>?K 9< JG<:@8C 89FLK K?<DgN?8K @EE<IHL8C@K@<J K?<P D@>?K ?8M< @E :FDDFE 9<PFE; K?< >C@KK<I PFL =@IJKEFK@:<; E; K?<E JFD<FE< <CJ< :FD<J 8CFE> 8E; J8PJ `2?P EFK:8CC this G<99C< 8 ;@8DFE; KFF E; K?@J FE< 8E; K?@J FE<a /?<P 8I< <EK?LJ@8JK@: 8E; K?<P D<8E N<CC !FI @K J<<DJ LE;<DF:I8K@:8E; <O:CLJ@FE8IP 8E; <C@K@JK 8E; LE?FC@JK@: KF :8CC JFD< G<99C<J `;@8eDFE;Ja 8E; FK?<IJ EFK $K J<<DJ_ narrow-minded… @= PFLcCC G8I;FEK?<G?I8J<#8I;CP open, ?8I;CP embracing, ?8I;CP communal.

4FL D@>?K K?@EB @K GF<K@: KF >@M< FE< NFI; D8EP D<8E@E>J 8E;K?<I<9P JGI<8; J?8;<J F= :FEEFK8K@FE 8CC 8IFLE; LK <M<E GF<KJ @= K?<P 8I< >FF; GF<KJ DLJK C<8IE KF J<< K?< NFIC; GI<:@J<CP $K @J EFK<EFL>? KF :FDG8I< CFM< KF 8 =CFN<I #FK A<8CFLJ LE:FEJLDD8K<;CFM< @J EFK K?< J8D< 8J K?< CFM< F= 8 :FLGC< D8II@<; =FI ;<:8;<J $= 

http://lesswrong.com/lw/ic/the_virtue_of_narrowness/ 

http://yudkowsky.net/virtues/  http://yudkowsky.net/virtues/ 

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 123/309

 PFL E<<; 8 =CFN<I KF JPD9FC@Q< A<8CFLJ CFM< PFL DLJK >F @EKF K?< >8I;<E 8E; CFFB 8E; D8B< JL9KC< ;@JK@E:K@FEJg=@E; 8 =CFN<I N@K?8 ?<8;P J:<EK 8E; 8 9I@>?K :FCFI 8E; K?FIEJ M<E @= PFLI @EK<EK

@J KF J?8;< D<8E@E>J 8E; :8JK :FEEFK8K@FEJ PFL DLJK B<<G GI<:@J<KI8:BF=<O8:KCPN?@:?D<8E@E>JPFLJ?8;<8E;:FEEFK<

$K @J 8 E<:<JJ8IP G8IK F= K?< I8K@FE8C@JKcJ 8IKgFI <M<E K?< GF<KcJ8IKgKF =F:LJ E8IIFNCP FE LELJL8C G<99C<J N?@:? GFJJ<JJ JFD< JG<e:@8C HL8C@KP E; CFFB 8K K?< ;<K8@CJ N?@:? K?FJ< G<99C<Jg8E; K?FJ<G<99C<J8CFE<gJ?8I<8DFE><8:?FK?<I /?@J@JEFK8J@E

$K @J G<I=<:KCP 8CC I@>?K =FI DF;<IE <MFCLK@FE8IP 9@FCF>@JKJ KF <OeGC8@E just  K?< G8KK<IEJ F= C@M@E> :I<8KLI<J 8E; EFK K?< `<MFCLK@FEaF= JK8IJ FI K?< `<MFCLK@FEa F= K<:?EFCF>P C8J JFD< LE=FIKLE8K<

JFLCJ LJ< K?< J8D< NFI; `<MFCLK@FEa KF :FM<I K?< E8KLI8CCP J<C<:K<;G8KK<IEJ F= I<GC@:8K@E> C@=< and  K?< JKI@:KCP 8::@;<EK8C JKIL:KLI< F= JK8IJ and K?< @EK<CC@><EKCP :FE=@>LI<; JKIL:KLI< F= K<:?EFCF>P E;8J N< 8CC BEFN @= G<FGC< LJ< K?< J8D< NFI; @K DLJK 8CC 9< K?< J8D<K?@E> 4FL J?FLC; 8LKFD8K@:8CCP ><E<I8C@Q< 8EPK?@E> PFL K?@EB PFLBEFN 89FLK 9@FCF>@:8C <MFCLK@FE KF K<:?EFCF>P EPFE< N?F K<CCJ

 PFL FK?<IN@J< DLJK 9< 8 D<I< GF@EKC<JJ G<;8EK $K :FLC;EcK GFJJ@9CP 9< K?8K PFLI 89PJD8C @>EFI8E:< F= DF;<IE <MFCLK@FE8IP K?<FIP @JJF KFK8C K?8K PFL :8EcK K<CC K?< ;@==<I<E:< 9<KN<<E 8 :8I9LI<KFI 8E;8 I8;@8KFI /?8KcJ LEK?@EB89C< )F K?< other >LPgPFL BEFN K?<FE< N?FcJ JKL;@<; K?< D8K?g@J ALJK KFF ;LD9 KF J<< K?< :FEE<:eK@FEJ

 E; N?8K :FLC; 9< DFI< M@IKLFLJ K?8E J<<@E> :FEE<:[email protected]<CP K?< N@J<JK F= 8CC ?LD8E 9<@E>J 8I< K?< )<N >< >LILJ N?FJ8P `M<IPK?@E> @J :FEE<:K<; KF <M<IPK?@E> <CJ<a $= PFL <M<I J8P K?@J 8CFL; PFL J?FLC; G8LJ< JF K?8K <M<IPFE< :8E 89JFI9 K?< J?<<IJ?F:BF=K?@J<<G2@J;FD

/?<I< @J 8 KI@M@8C D8GG@E> 9<KN<<E 8 >I8G? 8E; @KJ :FDGC<D<EK  =LCCP :FEE<:K<; >I8G? N@K? 8E <;>< 9<KN<<E <M<IP KNF M<IK@:<J:FEM<PJ K?< J8D< 8DFLEK F= @E=FID8K@FE 8J 8 >I8G? N@K? EF <;><J8K 8CC /?< @DGFIK8EK >I8G?J 8I< K?< FE<J N?<I< JFD< K?@E>J 8I< not :FEE<:K<;KFJFD<FK?<IK?@E>J

2?<E K?< LE<EC@>?K<E<; FE<J KIP KF 9< GIF=FLE; K?<P ;I8N <E;C<JJ M<I98C :FDG8I@JFEJ 9<KN<<E K?@J KFG@: 8E; K?8K KFG@:

 N?@:? @J C@B< K?@J N?@:? @J C@B< K?8K LEK@C K?<@I >I8G? @J =LCCP :FEeE<:K<; 8E; 8CJF KFK8CCP LJ<C<JJ /?< I<D<;P @J JG<:@=@: BEFNC<;><

,.!*,-'&**'/&++

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 124/309

8E; @Ee;<GK? JKL;P 2?<E PFL LE;<IJK8E; K?@E>J @E ;<K8@C PFL :8EJ<< ?FN K?<P 8I< not  8C@B< 8E; JK8IK <EK?LJ@8JK@:8CCP JL9KI8:K@E> <;><J off PFLI>I8G?

'@B<N@J< K?< @DGFIK8EK :8K<>FI@<J 8I< K?< FE<J K?8K ;F EFK :FEeK8@E <M<IPK?@E> @E K?< LE@M<IJ< "FF; ?PGFK?<J<J :8E FECP <OGC8@EJFD<GFJJ@9C<FLK:FD<J8E;EFKFK?<IJ

$K N8J G<I=<:KCP 8CC I@>?K =FI $J88: )<NKFE KF <OGC8@E just >I8M@KP just K?< N8P K?@E>J =8CC ;FNEg8E; ?FN GC8E<KJ FI9@K K?< .LE 8E;?FN K?< (FFE ><E<I8K<J K?< K@;<Jg9LK not K?< IFC< F= DFE<P @E ?LeD8E JF:@<KP FI ?FN K?< ?<8IK GLDGJ 9CFF; .E<<I@E> 8K E8IIFNE<JJ@J I8K?<I I<D@E@J:<EK F= 8E:@<EK "I<<BJ N?F K?FL>?K K?8K >F@E> FLK8E; 8:KL8CCP  looking  8K K?@E>J N8J D8EL8C C89FI 8E; D8EL8C C89FI

 N8J=FIJC8M<J J+C8KFGLK@Kh@E The Republic, Book VII  i

`$=8EPFE<J?FLC;K?IFN98:B?@J?<8;8E;C<8IEJFD<K?@E>9PJK8I@E>8KK?<M8I@<;G8KK<IEJFE8:<@C@E>8GG8I<EKCPPFLNFLC;K?@EBK?8K?<N8J:FEK<DGC8K@E> 

 N@K??@JI<8JFEN?<E?<N8JFECPJK8I@E>N@K??@J<P<J_$:8EEFK9LK9<C@<M<K?8KEFJKL;PD8B<JK?<JFLCCFFB

FE?@>?<O:<GKK?8KN?@:?@J:FE:<IE<;N@K?I<8C9<@E> 8E;K?<LEJ<<E2?<K?<I?<>8G<8E;JK8I<LGN8I;JFIJ?LK?@JDFLK?8E;JK8I<;FNEN8I;J@=@K9<K?@E>JF= K?<J<EJ<JK?8K?<KI@<JKFC<8IEJFD<K?@E>89FLK$;<:C8I<?<E<M<I:FLC;C<8IE=FIEFE<F=K?<J<K?@E>J8;D@KF=BEFNC<;><$J8P?@JJFLC@JCFFB@E>;FNEEFKLG<M<E@=?<@J=CF8K@E>FE?@J98:BFEC8E;FIFEJ<8a

(8EP KF;8P D8B< 8 J@D@C8I D@JK8B< 8E; K?@EB K?8K E8IIFN :FEe

:<GKJ 8I< 8J CFNCP 8E; LECF=KP 8E; LEG?@CFJFG?@:8C 8J J8P >F@E> FLK8E; CFFB@E> 8K K?@E>Jg8E <E;<8MFI FECP JL@K<; KF K?< LE;<I:C8JJLK I8K@FE8C@JKJg8E; 8CJF GF<KJgE<<; E8IIFN NFI;J KF <OGI<JJGI<:@J< K?FL>?KJ K?<P E<<; :8K<>FI@<J N?@:? @E:CL;< FECP JFD<K?@E>J 8E; <O:CL;< FK?<IJ /?<I<cJ EFK?@E> NIFE> N@K? =F:LJ@E> 

 PFLI D@E; E8IIFN@E> PFLI :8K<>FI@<J <O:CL;@E> GFJJ@9@C@K@<J 8E;J?8IG<E@E> PFLI GIFGFJ@K@FEJ -<8CCP K?<I< @JEcK $= PFL D8B< PFLI

 NFI;J KFF 9IF8; PFL <E; LG N@K? JFD<K?@E> K?8K @JEcK KIL< 8E;;F<JEcK<M<ED8B<>FF;GF<KIP

%1+,*!'-+ &+/*+,'%1+,*!'-+)-+,!'&+

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 125/309

 And DON’T EVEN GET ME STARTED on people who thinkWikipedia is an “Artificial Intelligence”, the invention of LSD was a “Sin- gularity” or that corporations are “superintelligent”! 

,.!*,-'&**'/&++

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 126/309

8. Your Strength as a Rationalist

h/?< =FCCFN@E> ?8GG<E<; KF D< @E 8E $- :?8KIFFD CFE> <EFL>?

8>F K?8K $ N8J JK@CC ?8E>@E> 8IFLE; @E $- :?8KIFFDJ /@D< ?8J=LQQ<;K?<D<DFIP8E;DPI<GFIKD8P9<@DGI<:@J<i

.F K?<I< $ N8J @E 8E $- :?8KIFFD N?<E JFD<FE< I<GFIKJ K?8K8 =I@<E; F= ?@J E<<;J D<;@:8C 8;M@:< #@J =I@<E; J8PJ K?8K ?<cJ 9<<E?8M@E> JL;;<E :?<JK G8@EJ JF ?< :8CC<; 8E 8D9LC8E:< 8E; K?< 8De9LC8E:< J?FN<; LG 9LK K?< G8I8D<;@:J KFC; ?@D @K N8J EFK?@E> 8E;C<=K 8E; EFN K?< :?<JK G8@EJ 8I< ><KK@E> NFIJ< 2?8K J?FLC; ?@J=I@<E;;F

$ N8J :FE=LJ<; 9P K?@J JKFIP $ I<D<D9<I<; I<8;@E> 89FLK?FD<C<JJ G<FGC< @E )<N 4FIB N?F NFLC; :8CC 8D9LC8E:<J ALJK KF9< K8B<E JFD<GC8:< N8ID 8E; ?FN K?< G8I8D<;@:J 8CN8PJ ?8; KFK8B< K?<D KF K?< <D<I><E:P IFFD <M<E FE K?< K? @K<I8K@FE <e:8LJ< @= K?<P ;@;EcK K?< 8D9LC8E:< :FDG8EP :FLC; 9< JL<; =FI CFKJ8E; CFKJ F= DFE<P '@B<N@J< <D<I><E:P IFFDJ 8I< C<>8CCP F9C@>8K<;KF KI<8K 8EPFE< I<>8I;C<JJ F= 89@C@KP KF G8P hE; K?< ?FJG@K8C 89eJFI9J K?< :FJKJ N?@:? 8I< <EFIDFLJ JF ?FJG@K8CJ 8I< :CFJ@E> K?<@I<D<I><E:P IFFDJ_ $K D8B<J PFL NFE;<I N?8KcJ K?< GF@EK F= ?8Me

@E> <:FEFD@JKJ @= N<cI< ALJK >F@E> KF @>EFI< K?<Di .F $ ;@;EcK HL@K<LE;<IJK8E; ?FN K?< ;<J:I@9<; <M<EKJ :FLC; ?8M< ?8GG<E<; AnyoneI<GFIK@E> JL;;<E :?<JK G8@EJ J?FLC; ?8M< 9<<E ?8LC<; F== 9P 8E 8De9LC8E:<@EJK8EKCP

 E; K?@J @J N?<I< $ =<CC ;FNE 8J 8 I8K@FE8C@JK $ I<D<D9<I<; J<Me<I8C F::8J@FEJ N?<I< DP ;F:KFI NFLC; :FDGC<K<CP =8@C KF G8E@: 8KK?< I<GFIK F= JPDGKFDJ K?8K J<<D<; KF D< M<IP 8C8ID@E> E; K?<(<;@:8C JK89C@J?D<EK N8J 8CN8PJ I@>?K M<IP J@E>C< K@D< $ ?8;:?<JK G8@EJ DPJ<C= 8K FE< GF@EK 8E; K?< ;F:KFI G8K@<EKCP <OGC8@E<;

KF D< K?8K $ N8J ;<J:I@9@E> :?<JK DLJ:C< G8@E EFK 8 ?<8IK 8KK8:B.F $ J8@; @EKF K?< $- :?8EE<C `2<CC @= K?< G8I8D<;@:J KFC; PFLI=I@<E; @K N8J EFK?@E> @K DLJK really be EFK?@E>gK?<Pc; ?8M< ?8LC<;?@DF==@=K?<I<N8JK?<K@E@<JK:?8E:<F=J<I@FLJKIFL9C<a

/?LJ $ D8E8><; KF <OGC8@E K?< JKFIP N@K?@E DP <O@JK@E> DF;<CK?FL>?K?<=@KJK@CC=<CK8C@KKC<=FI:<;_

http://lesswrong.com/lw/if/your_strength_as_a_rationalist/ 

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 127/309

'8K<I FE K?< =<CCFN :FD<J 98:B @EKF K?< $- :?8KIFFD 8E;J8PJ ?@J =I@<E; D8;< K?< N?FC< K?@E> LG M@;<EKCP K?@J N8J EFK FE<F=?@JDFI<I<C@89C<=I@<E;J

$ J?FLC; ?8M< I<8C@Q<; G<I?8GJ K?8K 8E LEBEFNE 8:HL8@EK8E:<F= 8E 8:HL8@EK8E:< @E 8E $- :?8EE<C D@>?K 9< C<JJ I<C@89C< K?8E8 GL9C@J?<; AFLIE8C 8IK@:C< C8J 9<C@<= @J <8J@<I K?8E ;@J9<C@<= N<9<C@<M<@EJK@E:K@M<CP9LK;@J9<C@<=I<HL@I<J8:FEJ:@FLJ<==FIK

.F @EJK<8; 9P ;@EK F= D@>?KP JKI8@E@E> $ =FI:<; DP DF;<C F= I<e8C@KP KF <OGC8@E 8E 8EFD8CP K?8K never actually happened. E; $ knew?FN <D98II8JJ@E> K?@J N8J $ knew K?8K K?< LJ<=LCE<JJ F= 8 DF;<C @JEFK N?8K @K :8E <OGC8@E 9LK N?8K @K :8EcK ?PGFK?<J@J K?8K =FI9@;JEFK?@E> G<ID@KJ <M<IPK?@E> 8E; K?<I<9P =8@CJ KF :FEJKI8@E 8EK@:@G8eK@FE

4FLI JKI<E>K? 8J 8 I8K@FE8C@JK @J PFLI 89@C@KP KF 9< DFI< :FE=LJ<;9P =@:K@FE K?8E 9P I<8C@KP $= PFL 8I< <HL8CCP >FF; 8K <OGC8@E@E> 8EP FLK:FD<PFL?8M<Q<IFBEFNC<;><

2< 8I< 8CC N<8B =IFD K@D< KF K@D< K?< J8; G8IK @J K?8K $ could ?8M< 9<<E JKIFE><I $ ?8; 8CC K?< @E=FID8K@FE $ E<<;<; KF 8II@M< 8KK?< :FII<:K 8EJN<I $ <M<E noticed K?< GIF9C<D 8E; K?<E $ @>EFI<;@K (P=<<C@E>F=:FE=LJ@FEN8J8CL<8E;$K?I<NDPCL<8N8P

$ J?FLC; ?8M< G8@; DFI< 8KK<EK@FE KF K?8K J<EJ8K@FE F=  still feels a little forced. $KcJ FE< F= K?< DFJK @DGFIK8EK =<<C@E>J 8 KILK?J<<B<I:8E ?8M< 8 G8IK F= PFLI JKI<E>K? 8J 8 I8K@FE8C@JK $K @J 8 ;<J@>E =C8N @E ?LD8E :F>E@K@FE K?8K K?@J J<EJ8K@FE D8E@=<JKJ 8J 8 HL@<K JKI8@E @EK?< 98:B F= PFLI D@E; @EJK<8; F= 8 N8@C@E> 8C8ID J@I<E 8E; 8 >CFNe@E> E<FE J@>E I<8;@E> `$/#- 4*0- (*' $. !'. *-/#$../*-4$.2-*)"a

http://www.overcomingbias.com/2007/08/truth-bias.html  http://www.wjh.harvard.edu/%7Edtg/ 

Gilbert%20et%20al%20%28EVERYTHING%20YOU%20READ%29.pdf  +8><b(8B@E><C@<=J+8P-<EKh@EEK@:@G8K<;OG<I@<E:<Jic

1'-* +,*&,+* ,!'&$!+,

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 128/309

9. Absence of Evidence Is Evidence of Absence

!IFD-F9PE8N<JcJ Rational Choice in an Uncertain World 

+FJKe?F:=@KK@E>F=<M@;<E:<KF?PGFK?<J@JN8J@EMFCM<;@E8DFJK>I@<MFLJ:?8GK<I@E0E@K<;.K8K<J?@JKFIPK?<@EK<IED<EKF=%8G8E<J<eD<I@:8EJ8KK?<9<>@EE@E>F= K?<.<:FE;2FIC;28I 2?<E8C@=FIE@8>FM<IEFI8IC28II<EK<JK@=@<;9<=FI<8:FE>I<JJ@FE8C?<8I@E>@E.8E!I8E:@J:FFE!<9IL8IP8HL<JK@FE<IGF@EK<;FLKK?8KK?<I<?8;9<<EEFJ89FK8><FI8EPFK?<IKPG<F= <JG@FE8><9PK?<%8G8E<J<eD<I@:8EJLGKFK?8KK@D<28II<EI<JGFE;<;`$K8B<K?<M@<NK?8KK?@JC8:BjF= JL9M<IJ@M<8:K@M@KPk@JK?<DFJKFD@EFLJJ@>E@EFLI

 N?FC<J@KL8K@FE$K:FEM@E:<JD<DFI<K?8EG<I?8GJ8EP FK?<I=8:KFIK?8KK?<J89FK8><N<8I<KF><KK?<!@=K?FCLDE8:K@M@K@<J8I<KF><K8I<K@D<;ALJKC@B<+<8IC#8I9FIN8JK@D<;_$9<C@<M<N<8I<ALJK9<@E>CLCC<;@EKF8=8CJ<J<EJ<F=J<:LI@KPa

FEJ@;<I 28II<EcJ 8I>LD<EK =IFD 8 8P<J@8E G<IJG<:K@M<2?<E N< J<< <M@;<E:< ?PGFK?<J<J K?8K 8JJ@>E<; 8 higher C@B<C@?FF;KF K?8K <M@;<E:< >8@E GIF989@C@KP 8K K?< <OG<EJ< F= ?PGFK?<J<J K?8K8JJ@>E<; 8 lower C@B<C@?FF; KF K?< <M@;<E:< /?@J @J 8 G?<EFD<EFEF=  relative C@B<C@?FF;J 8E; relative GIF989@C@K@<J 4FL :8E 8JJ@>E 8 ?@>? C@B<C@?FF; KF K?< <M@;<E:< 8E; JK@CC CFJ< GIF989@C@KP D8JJ KFJFD< FK?<I ?PGFK?<J@J @= K?8K FK?<I ?PGFK?<J@J 8JJ@>EJ 8 C@B<C@?FF;K?8K@J<M<E?@>?<I

28II<E J<<DJ KF 9< 8I>L@E> K?8K >@M<E K?8K N< J<< EF J89FK8><K?@J confirms K?8K 8 !@=K? FCLDE <O@JKJ 4FL :FLC; 8I>L< K?8K 8 !@=K? FCLDE might  ;<C8P @KJ J89FK8>< LK K?< C@B<C@?FF; @J JK@CC?@>?<I K?8K K?< absence F= 8 !@=K? FCLDE NFLC; G<I=FID 8E 89J<E:<F=J89FK8><

'<K JK8E; =FI K?< F9J<IM8K@FE F= J89FK8>< # =FI K?< ?PeGFK?<J@J F= 8 %8G8E<J<eD<I@:8E !@=K? FCLDE 8E; # =FI K?<?PGFK?<J@J K?8K EF !@=K? FCLDE <O@JKJ 2?8K<M<I K?< C@B<C@?FF;

http://lesswrong.com/lw/ih/absence_of_evidence_is_evidence_of_absence/  +8><bE$EKL@K@M<OGC8E8K@FEF=8P<J/?<FI<Dc

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 129/309

K?8K 8 !@=K? FCLDE NFLC; ;F EF J89FK8>< K?< GIF989@C@KP +hR#i@K :8EEFK 9< 8J C8I>< 8J K?< C@B<C@?FF; K?8K EF !@=K? FCLDE ;F<J EFJ89FK8>< K?< GIF989@C@KP +hR#i .F F9J<IM@E> 8 C8:B F= J89FK8><

@E:I<8J<JK?<GIF989@C@KPK?8KEF!@=K?FCLDE<O@JKJ  C8:B F= J89FK8>< ;F<JEcK prove K?8K EF !@=K? FCLDE <O@JKJ

 9J<E:< F= proof @J EFK proof F= 89J<E:< $E CF>@: e ` @DGC@<Ja@JEFK<HL@M8C<EKKFSeS`EFKe@DGC@<JEFKea

LK @E GIF989@C@KP K?<FIP 89J<E:< F=  evidence @J 8CN8PJ evidenceF= 89J<E:< $= @J 8 9@E8IP <M<EK 8E; +h#Ri +h#i `J<<@E> @E:I<8J<J K?< GIF989@C@KP F= #a K?<E +h#RSi +h#i `=8@CLI< KF F9eJ<IM< ;<:I<8J<J K?< GIF989@C@KP F= #a +h#i @J 8 N<@>?K<; D@O F= +h#Ri 8E; +h#RSi 8E; E<:<JJ8I@CP C@<J 9<KN<<E K?< KNF $= 

8EP F= K?@J JFLE;J 8K 8CC :FE=LJ@E> J<< E $EKL@K@M< OGC8E8K@FE F= 8P<J@8E-<8JFE@E> 

0E;<I K?< M8JK D8AFI@KP F= I<8CeC@=< :@I:LDJK8E:<J 8 :8LJ< D8P EFK I<C@89CP GIF;L:< J@>EJ F= @KJ<C= 9LK K?< 89J<E:< F= K?< :8LJ<@J <M<E C<JJ C@B<CP KF GIF;L:< K?< J@>EJ /?< 89J<E:< F= 8E F9J<Ie

 M8K@FE D8P 9< JKIFE> <M@;<E:< F= 89J<E:< FI M<IP N<8B <M@;<E:<F= 89J<E:< ;<G<E;@E> FE ?FN C@B<CP K?< :8LJ< @J KF GIF;L:< K?<F9J<IM8K@FE /?< 89J<E:< F= 8E F9J<IM8K@FE K?8K @J FECP N<8BCP G<Ie

D@KK<; h<M<E @= K?< 8CK<IE8K@M< ?PGFK?<J@J ;F<J EFK 8CCFN @K 8K 8CCi @J M<IP N<8B <M@;<E:< F= 89J<E:< hK?FL>? @K @J <M@;<E:< EFE<K?<C<JJi/?@J @J K?< =8CC8:P F= `>8GJ @E K?< =FJJ@C I<:FI;ag=FJJ@CJ =FID FECP I8I<CP @K @J =LK@C< KF KILDG<K K?< 89J<E:< F= 8 N<8BCP G<ID@KK<; F9eJ<IM8K@FE N?<E D8EP JKIFE> GFJ@K@M< F9J<IM8K@FEJ ?8M< 8CI<8;P 9<<EI<:FI;<; LK @= K?<I< 8I< no GFJ@K@M< F9J<IM8K@FEJ 8K 8CC @K @J K@D<KFNFIIP?<E:<K?<!<ID@+8I8;FO

4FLI JKI<E>K? 8J 8 I8K@FE8C@JK @J PFLI 89@C@KP KF 9< DFI< :FEe=LJ<; 9P =@:K@FE K?8E 9P I<8C@KP @= PFL 8I< <HL8CCP >FF; 8K <OGC8@E@E> 

8EP FLK:FD< PFL ?8M< Q<IF BEFNC<;>< /?< JKI<E>K? F= 8 DF;<C @JEFK N?8K @K can <OGC8@E 9LK N?8K @K can’t  =FI FECP GIF?@9@K@FEJ :FEeJKI8@E 8EK@:@G8K@FE $= PFL ;FEcK EFK@:< N?<E PFLI DF;<C D8B<JK?< <M@;<E:< LEC@B<CP PFL D@>?K 8J N<CC ?8M< EF DF;<C 8E; 8CJF PFLD@>?K8JN<CC?8M<EF<M@;<E:<EF9I8@E8E;EF<P<J

+8><bE$EKL@K@M<OGC8E8K@FEF=8P<J/?<FI<Dc

+8><b4FLI.KI<E>K?8J8-8K@FE8C@JKc +8><b<C@<=@E<C@<=c

+&'.!&!+.!&'+&

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 130/309

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 131/309

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 132/309

:FLEK<I<M@;<E:< F= 8 GFJJ@9C< `89EFID8Ca F9J<IM8K@FE PFL :8E FECP ;F @K 9P N<8B<E@E> K?< JLGGFIK F= 8 `EFID8Ca F9J<IM8K@FE KF 8 GI<e:@J<CP <HL8C 8E; FGGFJ@K< ;<>I<< $K @J 8 Q<IFeJLD >8D< )F D8KK<I

?FN PFL :FEE@M< EF D8KK<I ?FN PFL 8I>L< EF D8KK<I ?FN PFLJKI8K<>@Q< PFL :8EcK GFJJ@9CP <OG<:K K?< I<JLCK@E> >8D< GC8E KF J?@=K PFLI9<C@<=JhFE8M<I8><i@E8G8IK@:LC8I;@I<:K@FE

4FL D@>?K 8J N<CC J@K 98:B 8E; I<C8O N?@C< PFL N8@K =FI K?< <M@e;<E:<KF:FD<@E

_?LD8EGJP:?FCF>P@J so J:I<N<;LG

%1+,*!'-+ &+/*+,'%1+,*!'-+)-+,!'&+

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 133/309

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 134/309

Challenger <OGCF;<; =FI I<8JFEJ KI8:<; KF 8E *eI@E> CFJ@E> =C<O@9@C@KP 8K CFN K<DG<I8KLI< /?<I< N<I< N8IE@E> J@>EJ F= 8 GIF9C<D N@K?K?< *eI@E>J LK GI<M<EK@E> K?< Challenger  ;@J8JK<I NFLC; ?8M<

I<HL@I<; EFK 8KK<E;@E> KF K?< GIF9C<D N@K? K?< *eI@E>J 9LK 8KeK<E;@E> KF every N8IE@E> J@>E N?@:? J<<D<; 8J J<M<I< 8J K?< *eI@E> GIF9C<D without benefit of hindsight  $K :FLC; ?8M< 9<<E ;FE< 9LK @K

 NFLC; ?8M< I<HL@I<; 8 general policy DL:? DFI< <OG<EJ@M< K?8E ALJK=@O@E>K?<*e-@E>J

.?FIKCP 8=K<I .<GK<D9<I K? $ K?FL>?K KF DPJ<C= and now someone will turn up minor intelligence warnings of something-or-other, and then the hindsight will begin. 4<J $cD JLI< K?<P ?8; JFD< D@eEFI N8IE@E>J F= 8E 8C ,8<;8 GCFK 9LK K?<P GIF989CP 8CJF ?8; D@EFI

 N8IE@E>J F= D8=@8 8:K@M@KP EL:C<8I D8K<I@8C =FI J8C< 8E; 8E @EM8J@FE=IFD(8IJ

<:8LJ< N< ;FEcK J<< K?< :FJK F= 8 ><E<I8C GFC@:P N< C<8IE FM<ICP JG<:@=@: C<JJFEJ =K<I .<GK<D9<I K? K?< ! GIF?@9@K<; 9FOe:LKK<IJ FE 8@IGC8E<Jg8J @= K?< GIF9C<D ?8; 9<<E K?< =8@CLI< KF K8B<this particular `F9M@FLJa GI<:8LK@FE 2< ;FEcK C<8IE K?< ><E<I8C C<JeJFE the cost of effective caution is very high because you must attend to problems that are not as obvious now as past problems seem in hindsight.

/?< K<JK F= 8 DF;<C @J ?FN DL:? GIF989@C@KP @K 8JJ@>EJ KF K?<F9J<IM<; FLK:FD< #@E;J@>?K 9@8J JPJK<D8K@:8CCP ;@JKFIKJ K?@J K<JK N< K?@EB FLI DF;<C 8JJ@>E<; DL:? DFI< GIF989@C@KP K?8E @K 8:KL8CCP ;@; $EJKIL:K@E> K?< ALIP ;F<JEcK ?<CG 4FL ?8M< KF NI@K< ;FNE

 PFLIGI<;@:K@FEJ@E8;M8E:< *I8J!@J:??F==hiGLK@K

2?<EN<8KK<DGKKFLE;<IJK8E;G8JK<M<EKJN<@DGC@:@KCPK<JKK?<?PGFK?<J<JFIILC<JN<LJ<9FK?KF@EK<IGI<K8E;KF8EK@:@G8K<K?<NFIC;8IFLE;LJ$=@E

?@E;J@>?KN<JPJK<D8K@:8CCPLE;<I<JK@D8K<K?<JLIGI@J<JK?8KK?<G8JK?<C;8E;?FC;J=FILJN<8I<JL9A<:K@E> K?FJ<?PGFK?<J<JKF@EFI;@E8K<CPN<8BK<JKJ8E;GI<JLD89CP=@E;@E>C@KKC<I<8JFEKF:?8E><K?<D

+8><bFEJ<IM8K@FEF=OG<:K<;M@;<E:<c

%1+,*!'-+ &+/*+,'%1+,*!'-+)-+,!'&+

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 135/309

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 136/309

12. Hindsight Devalues Science

/?@J <O:<IGK =IFD (<P<IJcJ Exploring Social Psychology @J NFIK?

I<8;@E> @E <EK@I<KP LCC<E (LIG?P <;@KFI F= The Atlantic, J8@; K?8KK?< JF:@8C J:@<E:<J KLIE LG `EF @;<8J FI :FE:CLJ@FEJ K?8K :8EcK 9<=FLE; @E j8EPk <E:P:CFG<;@8 F= HLFK8K@FEJ_ 8P 8=K<I ;8P JF:@8CJ:@<EK@JKJ >F FLK @EKF K?< NFIC; 8P 8=K<I ;8P K?<P ;@J:FM<I K?8KG<FGC<cJ9<?8M@FI@JGI<KKPDL:?N?8KPFLc;<OG<:Ka

*= :FLIJ< K?< `<OG<:K8K@FEa @J 8CC ?@E;J@>?K h#@E;J@>?K [email protected]<:KJ N?F BEFN K?< 8:KL8C 8EJN<I KF 8 HL<JK@FE 8JJ@>E DL:??@>?<I GIF989@C@K@<J K?<P `NFLC; ?8M<a >L<JJ<; =FI K?8K 8EJN<I

:FDG8I<; KF JL9A<:KJ N?F DLJK >L<JJ N@K?FLK BEFN@E> K?< 8EJN<Ii/?< ?@JKFI@8E IK?LI .:?C<J@E><I %I ;@JD@JJ<; J:@<EK@=@: JKL;@<JF= 22$$ JFC;@<IJc <OG<I@<E:<J 8J `GFE;<IFLJ ;<DFEJKI8K@FEJa F= :FDDFEJ<EJ< !FI<O8DGC<

<KK<I<;L:8K<;JFC;@<IJJL==<I<;DFI<8;ALJKD<EKGIF9C<DJK?8EC<JJ<;L:8K<;JFC;@<IJh$EK<CC<:KL8CJN<I<C<JJGI<G8I<;=FI98KKC<JKI<JJ<JK?8EJKI<<KeJD8IKG<FGC<i

.FLK?<IEJFC;@<IJ:FG<;9<KK<IN@K?K?<?FK.FLK?.<8 

$JC8E;:C@D8K<K?8E)FIK?<IEJFC;@<IJh.FLK?<IE<IJ8I<DFI<8::LJKFD<;KF?FKN<8K?<Ii

2?@K<GI@M8K<JN<I<DFI<<8><IKF9<GIFDFK<;KFEFE:FDD@JJ@FE<;F==@:<IJK?8EC8:BGI@M8K<Jh4<8IJF= FGGI<JJ@FEK8B<8KFCCFE8:?@<M<D<EKDFK@M8K@FEi

.FLK?<IEC8:BJGI<=<II<;.FLK?<IEKF)FIK?<IE2?@K<F==@:<IJh9<:8LJ<.FLK?<IEF==@:<IJN<I<DFI<<OG<I@<E:<;8E;JB@CC<;@E@EK<I8:K@E>N@K?C8:BJi

JCFE>8JK?<=@>?K@E>:FEK@EL<;JFC;@<IJN<I<DFI<<8><IKFI<KLIE?FD<K?8E8=K<IK?<N8I<E;<;hLI@E> K?<=@>?K@E>JFC;@<IJBE<NK?<PN<I<@EDFIK8C;8E><Ii

#FN D8EP F= K?<J< =@E;@E>J ;F PFL K?@EB PFL could have GI<;@:Ke<; @E 8;M8E:< FLK F= FLK F= I< K?<I< 8EP :8J<J N?<I<

 PFL NFLC; ?8M< GI<;@:K<; K?< FGGFJ@K<gN?<I< PFLI DF;<C K8B<J 8 ?@K /8B<8DFD<EKKFK?@EB9<=FI<:FEK@EL@E>_

http://lesswrong.com/lw/im/hindsight_devalues_science/  http://csml.som.ohio-state.edu/Music829C/hindsight.bias.html 

+8><b#@E;J@>?K9@8Jc +8><bFEJ<IM8K@FEF=OG<:K<;M@;<E:<c

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 137/309

$E K?@J ;<DFEJKI8K@FE h=IFD +8LC '8Q8IJ=<C; 9P N8P F= (<P<IJi8CC F= K?< =@E;@E>J 89FM< 8I< K?< opposite F= N?8K N8J 8:KL8CCP =FLE;#FN D8EP K@D<J ;@; PFL K?@EB PFLI DF;<C KFFB 8 ?@K #FN D8EP 

K@D<J ;@; PFL 8;D@K PFL NFLC; ?8M< 9<<E NIFE> /?8KcJ ?FN >FF; PFLI DF;<C I<8CCP N8J /?< D<8JLI< F= PFLI JKI<E>K? 8J 8 I8K@FE8Ce@JK @JPFLI89@C@KPKF9<DFI<:FE=LJ<;9P=@:K@FEK?8E9PI<8C@KP

0EC<JJ F= :FLIJ< $ I<M<IJ<; K?< I<JLCKJ 8>8@E 2?8K ;F PFLK?@EB

F PFLI K?FL>?K GIF:<JJ<J 8K K?@J GF@EK N?<I< PFL really don’t BEFN K?< 8EJN<I =<<C ;@==<I<EK =IFD K?< K?FL>?K GIF:<JJ<J PFLLJ<;KFI8K@FE8C@Q<<@K?<IJ@;<F=K?<`BEFNEa8EJN<I

8G?E8 8I8KQ <OGFJ<; :FCC<>< JKL;<EKJ KF G8@IJ F= JLGGFJ<;=@E;@E>J FE< KIL< h`$E GIFJG<IFLJ K@D<J G<FGC< JG<E; 8 C8I><I GFIeK@FE F= K?<@I @E:FD< K?8E ;LI@E> 8 I<:<JJ@FEai 8E; FE< K?< KILK?cJFGGFJ@K< $E 9FK? J@;<J F= K?< G8@I JKL;<EKJ I8K<; K?< JLGGFJ<;=@E;@E> 8J N?8K K?<P `NFLC; ?8M< GI<;@:K<;a +<I=<:KCP JK8E;8I;?@E;J@>?K9@8J

2?@:? C<8;J G<FGC< KF K?@EB K?<P ?8M< EF E<<; =FI J:@<E:< 9<e:8LJ<K?<P`:FLC;?8M<GI<;@:K<;aK?8K

h%LJK8JPFLNFLC;<OG<:KI@>?Ki

#@E;J@>?K N@CC C<8; LJ KF JPJK<D8K@:8CCP LE;<IM8CL< K?< JLIGI@Je@E>E<JJ F= J:@<EK@=@: =@E;@E>J <JG<:@8CCP K?< ;@J:FM<I@<J N< under- stand  gK?< FE<J K?8K J<<D I<8C KF LJ K?< FE<J N< :8E I<KIF=@K @EKFFLI DF;<CJ F= K?< NFIC; $= PFL LE;<IJK8E; E<LIFCF>P FI G?PJ@:J8E; I<8; E<NJ @E K?8K KFG@: K?<E PFL GIF989CP LE;<I<JK@D8K< K?<JLIGI@J@E>E<JJ F= =@E;@E>J @E K?FJ< =@<C;J KFF /?@J LE=8@ICP ;<M8CeL<J K?< :FEKI@9LK@FE F= K?< I<J<8I:?<IJ 8E; NFIJ< N@CC GI<M<EK PFL=IFD EFK@:@E> N?<E PFL 8I< J<<@E> <M@;<E:< K?8K ;F<JEcK =@K  N?8K

 PFL really NFLC;?8M<<OG<:K<;2<E<<;KFD8B<8:FEJ:@FLJ<==FIKKF9<J?F:B<; enough.

+8><b4FLI.KI<E>K?8J8-8K@FE8C@JKc +8><bFEJ<IM8K@FEF=OG<:K<;M@;<E:<c

!&+!,.$-++!&

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 138/309

13. Fake Explanations

*E:< LGFE 8 K@D< K?<I< N8J 8E @EJKIL:KFI N?F K8L>?K G?PJ@:J JKLe

;<EKJ *E< ;8P J?< :8CC<; K?<D @EKF ?<I :C8JJ 8E; J?FN<; K?<D 8  N@;< JHL8I< GC8K< F= D<K8C E<OK KF 8 ?FK I8;@8KFI /?< JKL;<EKJ<8:? GLK K?<@I ?8E; FE K?< GC8K< 8E; =FLE; K?< J@;< E<OK KF K?<I8;@8KFI :FFC 8E; K?< ;@JK8EK J@;< N8ID E; K?< @EJKIL:KFI J8@;Why do you think this happens?  .FD< JKL;<EKJ >L<JJ<; :FEM<:K@FE F= 8@I :LII<EKJ 8E; FK?<IJ >L<JJ<; JKI8E>< D<K8CJ @E K?< GC8K< /?<P ;<M@J<; D8EP :I<8K@M< <OGC8E8K@FEJ EFE< JKFFG@E> JF CFN 8J KF J8P `$;FEcKBEFNaFI`/?@JJ<<DJ@DGFJJ@9C<a

 E; K?< 8EJN<I N8J K?8K 9<=FI< K?< JKL;<EKJ <EK<I<; K?< IFFDK?<@EJKIL:KFIKLIE<;K?<GC8K<8IFLE;

FEJ@;<I K?< JKL;<EK N?F =I8EK@:8CCP JK8DD<IJ `? D8P9< 9<e:8LJ< F= K?< ?<8K :FE;L:K@FE 8E; JFa $ 8JB @J K?@J 8EJN<I 8 GIFG<I9<C@<=  /?< NFI;J 8I< <8J@CP <EFL>? GIF=<JJ<; gJ8@; @E 8 CFL;<DG?8K@:MF@:< LK;FK?<NFI;J8:KL8CCP :FEKIFC8EK@:@G8K@FE

+FE;<I K?8K @EEF:<EK C@KKC< G?I8J< `9<:8LJ< F=a N?@:? :FD<J9<=FI< `?<8K :FE;L:K@FEa +FE;<I JFD< F= K?< other K?@E>J N< :FLC;GLK 8=K<I @K 2< :FLC; J8P =FI <O8DGC< `<:8LJ< F= G?CF>@JKFEa FI

`<:8LJ<F=D8>@:a`(8>@:a PFL :IP `/?8KcJ EFK 8 scientific <OGC8E8K@FEa $E;<<;

K?< G?I8J<J `9<:8LJ< F= ?<8K :FE;L:K@FEa 8E; `9<:8LJ< F= D8>@:a8I< I<8;@CP I<:F>E@Q<; 8J 9<CFE>@E> KF ;@==<I<EK literary genres. `#<8K:FE;L:K@FEa @J JFD<K?@E> K?8K .GF:B D@>?K J8P FE Star Trek N?<I<e8J`D8>@:aNFLC;9<J8@;9P"@C<J@E Buffy the Vampire Slayer 

#FN<M<I 8J 8P<J@8EJ N< K8B< EF EFK@:< F= C@K<I8IP ><EI<J !FILJ K?< JL9JK8E:< F= 8 DF;<C @J K?< :FEKIFC @K <O<IKJ FE 8EK@:@G8K@FE

$= PFL J8P `?<8K :FE;L:K@FEa N?8K <OG<I@<E:< ;F<J K?8K C<8; PFL KF anticipate?  0E;<I EFID8C :@I:LDJK8E:<J @K C<8;J PFL KF 8EK@:@G8K<K?8K @= PFL GLK PFLI ?8E; FE K?< J@;< F= K?< GC8K< E<8I K?< I8;@8eKFI K?8K J@;< N@CC =<<C N8ID<I K?8E K?< FGGFJ@K< J@;< $= `9<:8LJ< F= ?<8K :FE;L:K@FEa :8E 8CJF <OGC8@E K?< I8;@8KFIe8;A8:<EK J@;< =<<C@E> cooler, K?<E@K:8E<OGC8@EGI<KKPDL:? anything.

http://lesswrong.com/lw/ip/fake_explanations/  +8><b4FLI.KI<E>K?8J8-8K@FE8C@JKc +8><b<C@<=8JKK@I<c

+8><b+IF=<JJ@E>8E;?<<I@E>c +8><b<C@<=@E<C@<=c

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 139/309

 E; 8J  N< 8CC BEFN  9P  K?@J GF@EK h$ ;F ?FG< i@= PFL 8I< <HL8CCP >FF; 8K <OGC8@E@E> 8EP FLK:FD< PFL ?8M< Q<IFBEFNC<;>< `<:8LJ< F= ?<8K :FE;L:K@FEa LJ<; @E JL:? =8J?@FE @J

8 ;@J>L@J<; ?PGFK?<J@J F= D8O@DLD <EKIFGP $K @J 8EK@:@G8K@FEe@JFeDFIG?@: KF J8P@E> `D8>@:a $K =<<CJ C@B< 8E <OGC8E8K@FE 9LK @KcJ EFK

.LGGFJ<; K?8K @EJK<8; F= >L<JJ@E> N< D<8JLI<; K?< ?<8K F= K?<D<K8C GC8K< 8K M8I@FLJ GF@EKJ 8E; M8I@FLJ K@D<J .<<@E> 8 D<K8CGC8K< E<OK KF K?< I8;@8KFI N< NFLC; FI;@E8I@CP <OG<:K K?< GF@EKK<DG<I8KLI<J KF J8K@J=P 8E <HL@C@9I@LD F= K?< ;@==LJ@FE <HL8K@FE

 N@K? I<JG<:K KF K?< 9FLE;8IP :FE;@K@FEJ @DGFJ<; 9P K?< <EM@IFEeD<EK 4FL D@>?K EFK BEFN K?< <O8:K K<DG<I8KLI< F= K?< =@IJK GF@EKD<8JLI<; 9LK 8=K<I D<8JLI@E> K?< =@IJK GF@EKJg$cD EFK G?PJ@:@JK

<EFL>? KF BEFN ?FN D8EP NFLC; 9< I<HL@I<;gPFL :FLC; K8B< 8E<O:<CC<EK>L<JJ8KK?<I<JK

  KIL< D8JK<I F= K?< 8IK F= LJ@E> ELD9<IJ KF :FEJKI8@E K?< 8EeK@:@G8K@FE F= D8K<I@8C G?<EFD<E8g8 `G?PJ@:@JKagNFLC; K8B< JFD<D<8JLI<D<EKJ 8E; J8P `/?@J GC8K< N8J @E <HL@C@9I@LD N@K? K?< <Ee

 M@IFED<EK KNF 8E; 8 ?8C= D@ELK<J 8>F KLIE<; 8IFLE; 8E; @J EFN 8GGIF8:?@E><HL@C@9I@LD8>8@Ea

/?< ;<<G<I <IIFI F= K?< JKL;<EKJ @J EFK J@DGCP K?8K K?<P =8@C<; KF

:FEJKI8@E 8EK@:@G8K@FE /?<@I ;<<G<I <IIFI @J K?8K K?<P K?FL>?K K?<P  N<I< ;F@E> G?PJ@:J /?<P J8@; K?< G?I8J< `9<:8LJ< F=a =FCCFN<; 9P K?< JFIK F= NFI;J .GF:B D@>?K J8P FE Star Trek, 8E; K?FL>?K K?<P K?<I<9P<EK<I<;K?<D8>@JK<I@LDF=J:@<E:<

)FK JF /?<P J@DGCP DFM<; K?<@I D8>@: =IFD FE< C@K<I8IP ><EI<KF8EFK?<I

+8><b(8B@E><C@<=J+8P-<EKh@EEK@:@G8K<;OG<I@<E:<Jic +8><b<C@<=@E<C@<=c +8><b8P<J@8E%L;Fc +8><b+IF=<JJ@E>8E;?<<I@E>c

+8><b<C@<=8JKK@I<c http://lesswrong.com/lw/i8/religions_claim_to_be_nondisprovable/  +8><b!F:LJ4FLI0E:<IK8@EKPc +8><b4FLI.KI<E>K?8J8-8K@FE8C@JKc +8><b9J<E:<F=M@;<E:<$JM@;<E:<F=9J<E:<c

+8><b#@E;J@>?K<M8CL<J.:@<E:<c +8><bFEJ<IM8K@FEF=OG<:K<;M@;<E:<c

#0($&,!'&+

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 140/309

14. Guessing the Teacher’s Password

Followup to: !8B<OGC8E8K@FEJ

2?<E $ N8J PFLE> $ I<8; GFGLC8I G?PJ@:J 9FFBJ JL:? 8J-@:?8I; !<PED8EcJ QED: The Strange Theory of Light and Matter. $BE<N K?8K C@>?K N8J N8M<J JFLE; N8J N8M<J D8KK<I N8J N8M<J $KFFBGI@;<@EDPJ:@<EK@=@:C@K<I8:PN?<E$N8JE@E<P<8IJFC;

2?<E $ N8J FC;<I 8E; $ 9<>8E KF I<8; K?< Feynman Lectures on Physics, $ I8E 8:IFJJ 8 ><D :8CC<; `K?< N8M< <HL8K@FEa $ :FLC; =FCCFN K?< <HL8K@FEcJ ;<I@M8K@FE 9LK CFFB@E> 98:B $ :FLC;EcK J<< @KJ KILK?8K 8 >C8E:< .F $ K?FL>?K 89FLK K?< N8M< <HL8K@FE =FI K?I<< ;8PJ FE

8E; F== LEK@C $ J8N K?8K @K N8J <D98II8JJ@E>CP F9M@FLJ E; N?<E$ =@E8CCP LE;<IJKFF; $ I<8C@Q<; K?8K K?< N?FC< K@D< $ ?8; 8::<GK<;K?< ?FE<JK 8JJLI8E:< F= G?PJ@:@JKJ K?8K C@>?K N8J N8M<J JFLE; N8J

 N8M<J D8KK<I N8J N8M<J $ ?8; EFK ?8; K?< M8>L<JK @;<8 F= N?8K K?< NFI;`N8M<aD<8EKKF8G?PJ@:@JK

/?<I< @J 8E @EJK@E:K@M< K<E;<E:P KF K?@EB K?8K @= 8 G?PJ@:@JK J8PJ`C@>?K @J D8;< F= N8M<Ja 8E; K?< K<8:?<I J8PJ `2?8K @J C@>?K D8;<F=a 8E; K?< JKL;<EK J8PJ `28M<Ja K?< JKL;<EK ?8J D8;< 8 KIL<JK8K<D<EK /?8KcJ FECP =8@I I@>?K 2< 8::<GK `N8M<Ja 8J 8 :FII<:K

8EJN<I =IFD K?< G?PJ@:@JK NFLC;EcK @K 9< LE=8@I KF I<A<:K @K =IFD K?<JKL;<EK .LI<CPK?<8EJN<I`28M<Ja@J<@K?<I true FI false, I@>?K

2?@:? @J FE< DFI< 98; ?89@K KF LEC<8IE =IFD J:?FFC 2FI;J;F EFK ?8M< @EKI@EJ@: ;<=@E@K@FEJ $= $ ?<8I K?< JPCC89C<J `9<8eM<Ia8E; K?@EB F= 8 C8I>< IF;<EK K?8K @J 8 =8:K 89FLK DP FNE JK8K< F= D@E; EFK 8 =8:K 89FLK K?< JPCC89C<J `9<8eM<Ia /?< J<HL<E:< F= JPCeC89C<J `D8;< F= N8M<Ja hFI `9<:8LJ< F= ?<8K :FE;L:K@FE`i @J EFK 8  hypothesis, @K @J 8 G8KK<IE F= M@9I8K@FEJ KI8M<C@E> K?IFL>? K?< 8@I FI

@EB FE G8G<I $K :8E associate KF 8 ?PGFK?<J@J @E JFD<FE<cJ D@E;9LK @K @J EFK F= @KJ<C= I@>?K FI NIFE> LK @E J:?FFC K?< K<8:?<I?8E;J PFL 8 >FC; JK8I =FI saying `D8;< F= N8M<Ja N?@:? DLJK 9< K?<:FII<:K 8EJN<I 9<:8LJ< K?< K<8:?<I ?<8I; 8 G?PJ@:@JK <D@K K?< J8D<JFLE;eM@9I8K@FEJ .@E:< M<I98C 9<?8M@FI hJGFB<E FI NI@KK<Ei @J N?8K

http://lesswrong.com/lw/iq/guessing_the_teachers_password/  +8><b!8B<OGC8E8K@FEJc http://lesswrong.com/lw/hs/think_like_reality/  http://www.math.utah.edu/~pa/math/polya.html 

http://lesswrong.com/lw/i2/two_more_things_to_unlearn_from_school/  +8><b!8B<OGC8E8K@FEJc

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 141/309

 ><KJ K?< >FC; JK8I JKL;<EKJ 9<>@E KF K?@EB K?8K M<I98C 9<?8M@FI ?8J 8 KILK?eM8CL< =K<I 8CC <@K?<I C@>?K @J D8;< F= N8M<J FI @K @JEcK I@>?K

 E; K?@J C<8;J @EKF 8E <M<E NFIJ< ?89@K .LGGFJ< K?< K<8:?<I

GI<J<EKJ PFL N@K? 8 :FE=LJ@E> GIF9C<D @EMFCM@E> 8 D<K8C GC8K< E<OKKF 8 I8;@8KFI K?< =8I J@;< =<<CJ N8ID<I K?8E K?< J@;< E<OK KF K?< I8;@e8KFI /?< K<8:?<I 8JBJ `2?Pa $= PFL J8P `$ ;FEcK BEFNa PFL ?8M< no :?8E:< F= ><KK@E> 8 >FC; JK8Ig@K NFEcK <M<E :FLEK 8J :C8JJ G8IeK@:@G8K@FE LK ;LI@E> K?< :LII<EK J<D<JK<I K?@J K<8:?<I ?8J LJ<;K?< G?I8J<J `9<:8LJ< F= ?<8K :FEM<:K@FEa `9<:8LJ< F= ?<8K :FE;L:eK@FEa 8E; `9<:8LJ< F= I8;@8EK ?<8Ka *E< F= K?<J< @J GIF989CP N?8KK?< K<8:?<I N8EKJ 4FL J8P `? D8P9< 9<:8LJ< F= ?<8K :FE;L:eK@FEa

/?@J @J EFK 8 ?PGFK?<J@J about K?< D<K8C GC8K< /?@J @J EFK <M<E8 GIFG<I9<C@<=  $K@J8E8KK<DGKKF guess the teacher’s password.

M<E M@JL8C@Q@E> K?< JPD9FCJ F= K?< ;@==LJ@FE <HL8K@FE hK?< D8K? >FM<IE@E> ?<8K :FE;L:K@FEi ;F<JEcK D<8E PFLcM< =FID<; 8 ?PGFK?e<J@J about  K?< D<K8C GC8K< /?@J @J EFK J:?FFC N< 8I< EFK K<JK@E> 

 PFLI D<DFIP KF J<< @= PFL :8E NI@K< ;FNE K?< ;@==LJ@FE <HL8K@FE/?@J @J 8P<J:I8=K N< 8I< J:FI@E> PFLI 8EK@:@G8K@FEJ F= <OG<I@<E:<$= PFL use K?< ;@==LJ@FE <HL8K@FE 9P D<8JLI@E> 8 =<N GF@EKJ N@K? 8 

K?<IDFD<K<I 8E; K?<E KIP@E> KF GI<;@:K N?8K K?< K?<IDFD<K<I N@CCJ8P FE K?< E<OK D<8JLI<D<EK K?<E @K @J ;<=@E@K<CP :FEE<:K<; KF <OeG<I@<E:< M<E @= K?< JKL;<EK ALJK M@JL8C@Q<J JFD<K?@E> flowing, 8E;K?<I<=FI< ?FC;J 8 D8K:? E<8I K?< :FFC<I J@;< F= K?< GC8K< KF KIP KFD<8JLI< N?<I< K?< ?<8K >F<J K?<E K?@J D<EK8C @D8>< F= =CFN@E>eE<JJ:FEE<:KJKF<OG<I@<E:<@K:FEKIFCJ8EK@:@G8K@FE

$= PFL 8I<EcK using  K?< ;@==LJ@FE <HL8K@FEgGLKK@E> @E ELD9<IJ8E; ><KK@E> FLK I<JLCKJ K?8K :FEKIFC PFLI 8EK@:@G8K@FE F= G8IK@:LC8I<OG<I@<E:<JgK?<E K?< :FEE<:K@FE 9<KN<<E D8G 8E; K<II@KFIP @J

J<M<I<; 8J K?FL>? 9P 8 BE@=< 2?8K I<D8@EJ @J EFK 8 9<C@<=  9LK 8  M<I98C9<?8M@FI

$E K?< J:?FFC JPJK<D @KcJ 8CC 89FLK M<I98C 9<?8M@FI N?<K?<I NI@KK<EFE G8G<I FI JGFB<E 8CFL; 1<I98C 9<?8M@FI ><KJ PFL 8 >FC; JK8IFI 8 =8@C@E> >I8;< +8IK F= LEC<8IE@E> K?@J 98; ?89@K @J 9<:FD@E> 

+8><b!8B<OGC8E8K@FEJc

+8><b<C@<=8JKK@I<c +8><b<C@<=8JKK@I<c

-++!&,,* 6+(++/'*

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 142/309

:FEJ:@FLJCP 8N8I< F= K?< ;@==<I<E:< 9<KN<<E 8E <OGC8E8K@FE 8E; 8 G8JJNFI;

F<J K?@J J<<D KFF ?8IJ? 2?<E PFLcI< =8:<; 9P 8 :FE=LJ@E> 

D<K8C GC8K< :8EcK `#<8K :FE;L:K@FEa 9< 8 =@IJK JK<G KFN8I; =@E;@E> K?< 8EJN<I (8P9< 9LK FECP @= PFL ;FEcK =8CC @EKF K?< KI8G F= K?@EBe@E> K?8K PFL 8I< CFFB@E> =FI 8 G8JJNFI; 2?8K @= K?<I< @J EF K<8:?<IKF K<CC PFL K?8K PFL =8@C<; /?<E PFL D8P K?@EB K?8K `'@>?K @J

 N8B8C@O<Ja @J 8 >FF; <OGC8E8K@FE K?8K `N8B8C@O<Ja @J K?< :FII<:KG8JJNFI; $K ?8GG<E<; KF D< N?<E $ N8J E@E< P<8IJ FC;gEFK 9<e:8LJ< $ N8J JKLG@; 9LK 9<:8LJ< K?@J @J N?8K ?8GG<EJ by default. /?@J@J ?FN ?LD8E 9<@E>J K?@EB LEC<JJ K?<P 8I< KI8@E<; not  KF =8CC @EKFK?< KI8G #LD8E@KP JK8P<; JKL:B @E ?FC<J C@B< K?@J =FI K?FLJ8E;J F= 

 P<8IJ(8P9< @= N< ;I@CC JKL;<EKJ K?8K words don’t count, only anticipation-

controllers, K?< JKL;<EK N@CC not ><K JKL:B FE `#<8K :FE;L:K@FE )F(8P9< ?<8K :FEM<:K@FE /?8KcJ EFK @K <@K?<Ia (8P9< then, K?@EBe@E> K?< G?I8J< `#<8K :FE;L:K@FEa N@CC C<8; FEKF 8 ><EL@E<CP ?<CG=LCG8K?C@B<

[ `#<8K:FE;L:K@FEa[ LKK?8KcJFECP8G?I8J<gN?8K;F<J@KD<8E

[ /?<;@==LJ@FE<HL8K@FE[ LKK?FJ<8I<FECPJPD9FCJg?FN;F$8GGCPK?<D[ 2?8K;F<J8GGCP@E>K?<;@==LJ@FE<HL8K@FEC<8;D<KF

8EK@:@G8K<[ $KJLI<;F<JEcKC<8;D<KF8EK@:@G8K<K?8KK?<J@;<F=8 

D<K8CGC8K<=8IK?<I8N8P=IFD8I8;@8KFINFLC;=<<C N8ID<I

[ $ EFK@:< K?8K$8D :FE=LJ<; (8P9<K?<E<8IJ@;<ALJK feels :FFC<I9<:8LJ<@KcJD8;<F=DFI<@EJLC8K@M<D8K<I@8C

8E;KI8EJ=<IJC<JJ?<8KKFDP?8E; $cCCKIPD<8JLI@E>K?<K<DG<I8KLI<_

[ *B8PK?8KN8JEcK@K 8E$KIPKFM<I@=PN?<K?<IK?<;@==LJ@FE<HL8K@FE?FC;JKIL<F=K?@JD<K8CGC8K<8K8CC $J?<8K flowing K?<N8P@KLJL8CCP;F<JFI@JJFD<K?@E><CJ<

 >F@E>FE[ $:FLC;?FC;8D8K:?KFK?<GC8K<8E;KIPKFD<8JLI<?FN 

?<8KJGI<8;JFM<IK@D<_

+8><b4FLI.KI<E>K?8J8-8K@FE8C@JKc +8><b#@E;J@>?K<M8CL<J.:@<E:<c

%1+,*!'-+ &+/*+,'%1+,*!'-+)-+,!'&+

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 143/309

$= N< 8I< not  JKI@:K 89FLK `? D8P9< 9<:8LJ< F= ?<8K :FE;L:eK@FEa 9<@E> 8 =8B< <OGC8E8K@FE K?< JKL;<EK N@CC M<IP GIF989CP ><KJKL:B FE JFD< N8B8C@O<JeG8JJNFI; This happens by default, it hap-

 pened to the whole human species for thousands of years.

-++!&,,* 6+(++/'*

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 144/309

15. Science as Attire

.D8CC<IJKFID7Prerequisites: !8B<

OGC8E8K@FEJ <C@<= J KK@I<

/?< GI<M@<N =FI K?< X-Men DFM@< ?8J 8 MF@:<eFM<I J8P@E> `$E <M<IP ?LD8E 9<@E>_ K?<I< @JK?< ><E<K@: :F;<_ =FIDLK8K@FEa GG8I<EKCP 

 PFL :8E 8:HL@I< 8CC JFIKJ F= E<8K 89@C@K@<J 9P DLK8K@FE /?< DLK8EK.KFID=FI<O8DGC<?8JK?<89@C@KPKFK?IFNC@>?KE@E>9FCKJ

$ 9<> PFL ;<8I I<8;<I KF :FEJ@;<I K?< 9@FCF>@:8C D8:?@E<IP E<:e<JJ8IP KF ><E<I8K< <C<:KI@:@KP K?< 9@FCF>@:8C 8;8GK8K@FEJ E<:<JJ8IP KF 8MF@; 9<@E> ?8ID<; 9P <C<:KI@:@KP 8E; K?< :F>E@K@M< :@I:L@KIP I<eHL@I<; =FI =@E<CP KLE<; :FEKIFC F= C@>?KE@E> 9FCKJ $= N< 8:KL8CCP F9J<IM<; 8EP FI>8E@JD 8:HL@I@E> K?<J< 89@C@K@<J in one generation, 8JK?< I<JLCK F=  mutation, @K NFLC; FLKI@>?K =8CJ@=P K?< E<Fe8IN@E@8EDF;<C F= E8KLI8C J<C<:K@FE $K NFLC; 9< NFIJ< K?8E =@E;@E> I899@K

=FJJ@CJ @E K?< GI<e8D9I@8E $= <MFCLK@FE8IP K?<FIP :FLC; actuallyJKI<K:? KF :FM<I .KFID @K NFLC; 9< 89C< KF <OGC8@E 8EPK?@E>  8E;

 N<8CCBEFNN?8KK?8KNFLC;@DGCP

/?< X-Men :FD@:J LJ< K<IDJ C@B< `<MFCLK@FEa `DLK8K@FEa 8E;`><E<K@: :F;<a GLI<CP KF GC8:< K?<DJ<CM<J @E N?8K K?<P :FE:<@M< KF9< K?< literary genre F= J:@<E:< /?< G8IK K?8K J:8I<J D< @J NFE;<Ie@E> ?FN D8EP G<FGC< <JG<:@8CCP @E K?< D<;@8 LE;<IJK8E; J:@<E:<only 8J8C@K<I8IP><EI<

$ <E:FLEK<I G<FGC< N?F M<IP ;<=@E@K<CP 9<C@<M< @E

<MFCLK@FE N?F JE<<I 8K K?< =FCCP F= :I<8K@FE@JKJ E; P<K K?<P ?8M< EF @;<8 F= N?8K K?< K?<FIP F= <MFCLK@FE8IP 9@FCF>P G<ID@KJ 8E; GIF?@9@KJ/?<PcCC K8CB 89FLK `K?< E<OK JK<G @E K?< <MFCLK@FE F= ?LD8E@KPa 8J@= E8KLI8C J<C<:K@FE >FK ?<I< 9P =FCCFN@E> 8 GC8E *I <M<E NFIJ<K?<PcCC K8CB 89FLK JFD<K?@E> :FDGC<K<CP FLKJ@;< K?< ;FD8@E F= <MFe

http://lesswrong.com/lw/ir/science_as_attire/  +8><b!8B<OGC8E8K@FEJc +8><b<C@<=8JKK@I<c

+8><b4FLI.KI<E>K?8J8-8K@FE8C@JKc +8><b+IF=<JJ@E>8E;?<<I@E>c

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 145/309

CLK@FE8IP 9@FCF>P C@B< 8E @DGIFM<; ;<J@>E =FI :FDGLK<I :?@GJ FI:FIGFI8K@FEJ JGC@KK@E> FI ?LD8EJ LGCF8;@E> K?<DJ<CM<J @EKF :FDeGLK<IJ 8E; K?<PcCC :8CC that  `<MFCLK@FEa $= <MFCLK@FE8IP 9@FCF>P 

:FLC;:FM<IK?8K@K:FLC;:FM<I8EPK?@E>+IF989CP 8E 8:KL8C D8AFI@KP F= K?< G<FGC< N?F believe in <MFCLe

K@FE LJ< K?< G?I8J< `9<:8LJ< F= <MFCLK@FEa 9<:8LJ< K?<P N8EK KF 9<G8IK F= K?< J:@<EK@=@: @Ee:IFN;g9<C@<= 8J J:@<EK@=@: 8KK@I< C@B< N<8Ie@E> 8 C89 :F8K $= K?< J:@<EK@=@: @Ee:IFN; @EJK<8; LJ<; K?< G?I8J<`9<:8LJ< F= @EK<CC@><EK ;<J@>Ea K?<P NFLC; ALJK 8J :?<<I=LCCP LJ<K?8K @EJK<8;g@K NFLC; D8B< EF ;@==<I<E:< KF K?<@I 8EK@:@G8K@FEe:FEKIFCC<IJ .8P@E> `9<:8LJ< F= <MFCLK@FEa @EJK<8; F= `9<:8LJ< F= @EK<CC@><EK ;<J@>Ea ;F<J EFK for them, GIF?@9@K .KFID $KJ FECP GLIeGFJ<=FIK?<D@JKF@;<EK@=PN@K?8KI@9<

$ <E:FLEK<I G<FGC< N?F 8I< HL@K< N@CC@E> KF <EK<IK8@E K?< EFK@FEF= ;LD9<IeK?8Ee?LD8E IK@=@:@8C $EK<CC@><E:< FI <M<E D@C;CP JD8IK<IeK?8Ee?LD8E IK@=@:@8C $EK<CC@><E:< $EKIF;L:< K?< EFK@FEF= JKIFE>CP JLG<I?LD8E IK@=@:@8C $EK<CC@><E:< 8E; K?<PcCC JL;;<ECP ;<:@;< @KcJ `GJ<L;FJ:@<E:<` $KcJ EFK K?8K K?<P K?@EB K?<P ?8M< 8 K?<FIP F= @EK<CC@><E:< N?@:? C<KJ K?<D :8C:LC8K< 8 K?<FI<K@:8C LGG<I9FLE; FE K?< GFN<I F= 8E FGK@D@Q8K@FE GIF:<JJ -8K?<I K?<P 8Je

JF:@8K< JKIFE>CP JLG<I?LD8E $ KF K?< literary genre F= 8GF:8CPGK@:C@K<I8KLI< N?<I<8J 8E $ ILEE@E> 8 JD8CC :FIGFI8K@FE 8JJF:@8K<J KFK?< C@K<I8IP ><EI< F=  Wired  D8>8Q@E< /?<P 8I<EcK JG<8B@E> =IFD

 N@K?@E 8 DF;<C F= :F>E@K@FE /?<P ;FEcK I<8C@Q< K?<P need 8 DF;<C/?<P ;FEcK I<8C@Q< K?8K J:@<E:< @J about DF;<CJ /?<@I ;<M8JK8K@E> :I@K@HL<J :FEJ@JK GLI<CP F= comparisons to apocalyptic literature I8K?<IK?8E J8P BEFNE C8NJ N?@:? GIF?@9@K JL:? 8E FLK:FD< /?<P LEe;<IJK8E; J:@<E:< only 8J 8 C@K<I8IP ><EI< FI @Ee>IFLG KF 9<CFE> KF/?< 8KK@I< ;F<JEcK CFFB KF K?<D C@B< 8 C89 :F8K K?@J @JEcK K?< =FFKe

98CCK<8DK?<PcI< :?<<I@E>  =FI$J K?<I< 8EPK?@E> @E J:@<E:< K?8K PFL 8I< proud F= 9<C@<M@E> 8E;

 P<K PFL ;F EFK LJ< K?< 9<C@<= GIF=<JJ@FE8CCP 4FL ?8; 9<JK 8JB PFLIeJ<C= N?@:? =LKLI< <OG<I@<E:<J PFLI 9<C@<=  prohibits =IFD ?8GG<E@E> KF PFL /?8K @J K?< JLD F= N?8K PFL ?8M< 8JJ@D@C8K<; 8E; D8;< 8 

+8><b!8B<OGC8E8K@FEJc +8><b<C@<=8JKK@I<c http://lesswrong.com/lw/io/is_molecular_nanotechnology_scientific/ 

+8><b<C@<=8JKK@I<c +8><b+IF=<JJ@E>8E;?<<I@E>c

+!&+,,!*

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 146/309

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 147/309

16. Fake Causality 

Followup to !8B< OGC8E8K@FEJ "L<JJ@E> K?< /<8:?<IcJ +8JJe

 NFI;

+?CF>@JKFE N8J K?< :<EKLIPcJ 8EJN<I KF K?< C<D<EK8C !@I<F= K?< "I<<B 8C:?<D@JKJ $>E@K< NFF; 8E; C<K @K 9LIE 2?8K @JK?< FI8E><Pe9I@>?K `=@I<a JKL== 2?P ;F<J K?< NFF; KI8EJ=FID @EKF8J? /F 9FK? HL<JK@FEJ K?< K?e:<EKLIP :?<D@JKJ 8EJN<I<;`G?CF>@JKFEa

_8E;K?8KN8J@KPFLJ<<K?8KN8JK?<@I8EJN<I `+?CF>@JKFEa

+?CF>@JKFE <J:8G<; =IFD 9LIE@E> JL9JK8E:<J 8J M@J@9C< =@I< J

K?< G?CF>@JKFE <J:8G<; K?< 9LIE@E> JL9JK8E:<J CFJK G?CF>@JKFE 8E;JF 9<:8D< 8J? K?< `KIL< D8K<I@8Ca !C8D<J @E <E:CFJ<; :FEK8@E<IJ

 N<EK FLK 9<:8LJ< K?< 8@I 9<:8D< J8KLI8K<; N@K? G?CF>@JKFE 8E; JF:FLC; EFK ?FC; 8EP DFI< ?8I:F8C C<=K C@KKC< I<J@;L< LGFE 9LIE@E> 9<:8LJ<@KN8JE<8ICPGLI<G?CF>@JKFE

*= :FLIJ< FE< ;@;EcK LJ< G?CF>@JKFE K?<FIP KF predict K?< FLKe:FD< F= 8 :?<D@:8C KI8EJ=FID8K@FE 4FL CFFB<; 8K K?< I<JLCK =@IJKK?<E PFL LJ<; G?CF>@JKFE K?<FIP KF explain @K $KcJ EFK K?8K G?CFe

 >@JKFE K?<FI@JKJ GI<;@:K<; 8 =C8D< NFLC; <OK@E>L@J? @E 8 :CFJ<;:FEK8@E<I I8K?<I K?<P C@K 8 =C8D< @E 8 :FEK8@E<I N8K:?<; @K >F FLK8E; K?<E J8@; `/?< 8@I DLJK ?8M< 9<:FD< J8KLI8K<; N@K? G?CFe

 >@JKFEa 4FL :FLC;EcK <M<E LJ< G?CF>@JKFE K?<FIP KF J8P N?8K PFLFL>?K not KFJ<<@K:FLC;<OGC8@E<M<IPK?@E>

/?@J N8J 8E <8IC@<I 8>< F= J:@<E:< !FI 8 CFE> K@D< EF FE< I<8Ce@Q<; K?<I< N8J 8 GIF9C<D !8B< <OGC8E8K@FEJ ;FEcK feel =8B< /?8KcJ

 N?8KD8B<JK?<D;8E><IFLJ

(F;<IE I<J<8I:? JL>><JKJ K?8K ?LD8EJ K?@EB 89FLK :8LJ< 8E;

<==<:K LJ@E> JFD<K?@E> C@B< K?< ;@I<:K<; 8:P:C@: >I8G?J h"Ji F= 8P<J E<KJ <:8LJ< @K I8@E<; K?< J@;<N8CB @J N<K 9<:8LJ< K?< J@;<e

 N8CB@JN<K@K@JJC@GG<IP

j-8@Ekej.@;<N8CBN<Kkej.@;<N8CBJC@GG<IPk

http://lesswrong.com/lw/is/fake_causality/  +8><b!8B<OGC8E8K@FEJc +8><b"L<JJ@E>K?</<8:?<IcJ+8JJNFI;c

+8><b4FLI.KI<E>K?8J8-8K@FE8C@JKc +8><b!8B<OGC8E8K@FEJc

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 148/309

!IFD K?@J N< :8E @E=<IgFI @E 8 8P<J E<K I@>FIFLJCP :8C:LC8K<@E GIF989@C@K@<JgK?8K N?<E K?< J@;<N8CB @J JC@GG<IP @K GIF989CP I8@E<; 9LK @= N< 8CI<8;P BEFN K?8K K?< J@;<N8CB @J N<K C<8IE@E> 

K?8K K?< J@;<N8CB @J JC@GG<IP K<CCJ LJ EFK?@E> DFI< 89FLK N?<K?<I @KI8@E<;

2?P@J=@I<?FK8E;9I@>?KN?<E@K9LIEJ

j`+?CF>@JKFEakej!@I<?FK8E;9I@>?Kk

$K feels C@B< 8E <OGC8E8K@FE $KcJ represented LJ@E> K?< J8D< :F>eE@K@M< ;8K8 =FID8K LK K?< ?LD8E D@E; ;F<J EFK 8LKFD8K@:8CCP ;<K<:K N?<E 8 :8LJ< ?8J 8E LE:FEJKI8@E@E> 8IIFN KF @KJ <==<:K2FIJ< K?8EBJ KF ?@E;J@>?K 9@8J @K D8P =<<C C@B< K?< :8LJ< :FEe

JKI8@EJ

K?<<==<:KN?<E@KN8JD<I<CP =@KK<;

KFK?<<==<:K$EK<I<JK@E>CP FLI DF;<IE LE;<IJK8E;@E> F= GIF989@C@JK@: I<8eJFE@E> 89FLK :8LJ8C@KP  :8E ;<J:I@9< GI<:@J<CP N?8K K?< G?CF>@JKFEK?<FI@JKJ N<I< ;F@E> NIFE> *E< F= K?< GI@D8IP @EJG@I8K@FEJ =FI8P<J@8E E<KNFIBJ N8J EFK@:@E> K?< GIF9C<D F= ;FL9C<e:FLEK@E> <Me@;<E:< @= @E=<I<E:< I<JFE8K<J 9<KN<<E 8E <==<:K 8E; 8 :8LJ< !FI<O8DGC< C<KcJ J8P K?8K $ ><K 8 9@K F= LEI<C@89C< @E=FID8K@FE K?8K K?<J@;<N8CB @J N<K /?@J J?FLC; D8B< D< K?@EB @KcJ DFI< C@B<CP KF 9<I8@E@E> LK @= @KcJ DFI< C@B<CP KF 9< I8@E@E> ;F<JEcK K?8K D8B< @K

DFI< C@B<CP K?8K K?< J@;<N8CB @J N<K E; NFLC;EcK that  D8B< @KDFI< C@B<CP K?8K K?< J@;<N8CB @J JC@GG<IP LK @= K?< J@;<N8CB @J JC@GeG<IP @KcJ GIF989CP N<K 8E; K?<E $ J?FLC; 8>8@E I8@J< DP GIF989@C@KP K?8K@KcJI8@E@E>_

 %L;<8 +<8IC LJ<J K?< D<K8G?FI F= 8E 8C>FI@K?D =FI :FLEK@E> JFCe;@<IJ @E 8 C@E< .LGGFJ< PFLcI< @E K?< C@E< 8E; PFL J<< KNF JFC;@<IJE<OK KF PFL FE< @E =IFEK 8E; FE< @E 98:B /?8KcJ K?I<< JFC;@<IJ .F

 PFL 8JB K?< JFC;@<I E<OK KF PFL `#FN D8EP JFC;@<IJ ;F you J<<a#< CFFBJ 8IFLE; 8E; J8PJ `/?I<<a .F K?8KcJ 8 KFK8C F= J@O JFC;@<IJ/?@JF9M@FLJCP@J not ?FNKF;F@K

  JD8IK<I N8P @J KF 8JB K?< JFC;@<I @E =IFEK F= PFL `#FN D8EP JFC;@<IJ =FIN8I; F= PFLa 8E; K?< JFC;@<I @E 98:B `#FN D8EP JFC;@<IJ 98:BN8I; F= PFLa /?< HL<JK@FE `#FN D8EP JFC;@<IJ =FIe

+8><b#@E;J@>?K9@8Jc +8><b(8B@E><C@<=J+8P-<EKh@EEK@:@G8K<;OG<I@<E:<Jic +8><bFEJ<IM8K@FEF=OG<:K<;M@;<E:<c http://books.google.com/ 

books?id=k9VsqN24pNYC&dq=&pg=PP1&ots=WR9UGWdOdd&sig=w_Mrax-y4VVwZy5SQ-

GySphNsKMc&prev=http://www.google.com/  search%3Fhl%3Den%26safe%3Doff%26q%3Dpearl%2Bintelligent%2Bsystems%26btnG%3DSearch&sa=X&oi=print 

%1+,*!'-+ &+/*+,'%1+,*!'-+)-+,!'&+

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 149/309

 N8I;a :8E 9< G8JJ<; FE 8J 8 D<JJ8>< N@K?FLK :FE=LJ@FE $= $cD 8KK?< =IFEK F= K?< C@E< $ G8JJ K?< D<JJ8>< ` JFC;@<I =FIN8I;a =FI DPeJ<C= /?< G<IJFE ;@I<:KCP @E 98:B F= D< ><KJ K?< D<JJ8>< ` JFC;@<I

=FIN8I;a 8E; G8JJ<J FE K?< D<JJ8>< ` JFC;@<IJ =FIN8I;a KF K?< JFCe;@<I 9<?@E; ?@D K K?< J8D< K@D< <8:? JFC;@<I @J 8CJF ><KK@E> K?<D<JJ8>< `) JFC;@<IJ 98:BN8I;a =IFD K?< JFC;@<I 9<?@E; K?<D 8E;G8JJ@E> @K FE 8J `) JFC;@<IJ 98:BN8I;a KF K?< JFC;@<I @E =IFEK F= K?<D #FN D8EP JFC;@<IJ @E KFK8C ;; K?< KNF ELD9<IJ PFL I<e:<@M< GCLJ FE< =FI PFLIJ<C= K?8K @J K?< KFK8C ELD9<I F= JFC;@<IJ @EC@E<

/?< B<P @;<8 @J K?8K <M<IP JFC;@<I DLJK separately KI8:B K?<KNF D<JJ8><J K?< =FIN8I;eD<JJ8>< 8E; 98:BN8I;eD<JJ8>< 8E; 8;;

K?<D KF><K?<I FECP 8K K?< <E; 4FL E<M<I 8;; 8EP JFC;@<IJ =IFDK?< 98:BN8I;eD<JJ8>< PFL I<:<@M< KF K?< =FIN8I;eD<JJ8>< PFL G8JJ98:B $E;<<; K?< KFK8C ELD9<I F= JFC;@<IJ @J E<M<I G8JJ<; 8J 8 D<JeJ8><gEFFE<<M<IJ8PJ@K8CFL;

 E 8E8CF>FLJ GI@E:@GC< FG<I8K<J @E I@>FIFLJ GIF989@C@JK@: I<8eJFE@E> 89FLK :8LJ8C@KP $= PFL C<8IE JFD<K?@E> 89FLK N?<K?<I @KcJI8@E@E> =IFD JFD< JFLI:< other K?8E F9J<IM@E> K?< J@;<N8CB KF 9<

 N<K K?@J N@CC J<E; 8 =FIN8I;eD<JJ8>< =IFD jI8@Ek KF jJ@;<N8CB N<Kk8E; I8@J< FLI <OG<:K8K@FE F= K?< J@;<N8CB 9<@E> N<K $= PFL F9J<IM<K?< J@;<N8CB KF 9< N<K K?@J J<E;J 8 98:BN8I;eD<JJ8>< KF FLI 9<eC@<= K?8K @K @J I8@E@E> 8E; K?@J D<JJ8>< GIFG8>8K<J =IFD jI8@Ek KF 8CCE<@>?9FI@E> EF;<J except K?< jJ@;<N8CB N<Kk EF;< 2< :FLEK <8:?G@<:< F= <M@;<E:< <O8:KCP FE:< EF LG;8K< D<JJ8>< <M<I `9FLE:<Ja98:B 8E; =FIK? /?< <O8:K 8C>FI@K?D D8P 9< =FLE; @E %L;<8 +<8ICcJ:C8JJ@: `+IF989@C@JK@: -<8JFE@E> @E $EK<CC@><EK .PJK<DJ )<KNFIBJF=+C8LJ@9C<$E=<I<E:<`

.F N?8K N<EK NIFE> @E G?CF>@JKFE K?<FIP 2?<E N< F9J<IM<

K?8K =@I< @J ?FK K?< j=@I<k EF;< :8E J<E; 8 98:BN8I;e<M@;<E:< KF K?<j`G?CF>@JKFEak EF;< C<8;@E> LJ KF LG;8K< FLI 9<C@<=J 89FLK G?CF>@JeKFE LK @= JF N< :8EcK :FLEK K?@J 8J 8 JL::<JJ=LC =FIN8I;eGI<;@:K@FEF= G?CF>@JKFE K?<FIP /?< D<JJ8>< J?FLC; >F @E FECP FE< ;@I<:K@FE8E;EFK9FLE:<98:B

http://books.google.com/ books?id=k9VsqN24pNYC&dq=&pg=PP1&ots=WR9UGWdOdd&sig=w_Mrax-y4VVwZy5SQ-

GySphNsKMc&prev=http://www.google.com/  search%3Fhl%3Den%26safe%3Doff%26q%3Dpearl%2Bintelligent%2Bsystems%26btnG%3DSearch&sa=X&oi=print 

#-+$!,1

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 150/309

 C8J ?LD8E 9<@E>J ;F EFK LJ< 8 I@>FIFLJ 8C>FI@K?D =FI LG;8K@E> 9<C@<= E<KNFIBJ 2< C<8IE 89FLK G8I<EK EF;<J =IFD F9J<IM@E> :?@Ce;I<E 8E; GI<;@:K :?@C; EF;<J =IFD 9<C@<=J 89FLK G8I<EKJ LK N<

;FEcK B<<G I@>FIFLJCP J<G8I8K< 9FFBJ =FI K?< 98:BN8I;eD<JJ8>< 8E;=FIN8I;eD<JJ8>< 2< ALJK I<D<D9<I K?8K G?CF>@JKFE @J ?FK N?@:?causes =@I< KF 9< ?FK .F @K J<<DJ C@B< G?CF>@JKFE K?<FIP GI<;@:KJ K?<?FKE<JJ F= =@I< *I NFIJ< @K ALJK =<<CJ C@B< phlogiston makes the fire hot.

0EK@C PFL EFK@:< K?8K EF advance GI<;@:K@FEJ 8I< 9<@E> D8;< K?<EFEe:FEJKI8@E@E> :8LJ8C EF;< @J EFK C89<C<; `=8B<a $KcJ I<GI<J<EK<;K?< J8D< N8P 8J 8EP FK?<I EF;< @E PFLI 9<C@<= E<KNFIB $K =<<CJ C@B<8=8:KC@B<8CCK?<FK?<I=8:KJPFLBEFN Phlogiston makes the fire hot.

  GIFG<ICP ;<J@>E<; $ NFLC; EFK@:< K?< GIF9C<D @EJK8EKCP/?@J NFLC;EcK <M<E I<HL@I< JG<:@8CeGLIGFJ< :F;< ALJK :FII<:K 9FFBeB<<G@E> F= K?< 9<C@<= E<KNFIB h.8;CP N< ?LD8EJ :8EcK I<NI@K< FLIFNE:F;<K?<N8P8GIFG<ICP;<J@>E<;$:FLC;i

.G<8B@E> F= `?@E;J@>?K 9@8Ja @J ALJK K?< EFEK<:?E@:8C N8P F= J8P@E> K?8K ?LD8EJ ;F EFK I@>FIFLJCP J<G8I8K< =FIN8I; 8E; 98:Be

 N8I; D<JJ8><J 8CCFN@E> =FIN8I; D<JJ8><J KF 9< :FEK8D@E8K<; 9P 98:BN8I;FE<J

/?FJ< N?F CFE> 8>F N<EK ;FNE K?< G8K? F= G?CF>@JKFE N<I< EFKKIP@E> KF 9< =FFCJ )F J:@<EK@JK ;<C@9<I8K<CP N8EKJ KF ><K JKL:B @E 8 9C@E; 8CC<P I< K?<I< 8EP =8B< <OGC8E8K@FEJ @E your D@E; $= K?<I<8I< $ >L8I8EK<< K?<PcI< EFK C89<C<; `=8B< <OGC8E8K@FEa JF GFCC@E> 

 PFLIK?FL>?KJ=FIK?<`=8B<aB<PNFI;N@CCEFKKLIEK?<DLG

/?8EBJ KF ?@E;J@>?K 9@8J @KcJ 8CJF EFK <EFL>? KF :?<:B ?FN  N<CC PFLI K?<FIP `GI<;@:KJa =8:KJ PFL 8CI<8;P BEFN 4FLcM< >FK KFGI<;@:K =FI KFDFIIFN EFK P<JK<I;8P $KcJ K?< FECP N8P 8 D<JJP ?LeD8ED@E;:8E9<>L8I8EK<<;F=J<E;@E>8GLI<=FIN8I;D<JJ8><

+8><b#@E;J@>?K<M8CL<J.:@<E:<c +8><b#@E;J@>?K<M8CL<J.:@<E:<c

%1+,*!'-+ &+/*+,'%1+,*!'-+)-+,!'&+

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 151/309

17. Semantic Stopsigns

 And the child asked:

, 2?<I<;@;K?@JIF:B:FD<=IFD  $:?@GG<;@KF==K?<9@>9FLC;<I8KK?<:<EK<IF=K?<M@CC8><, 2?<I<;@;K?<9FLC;<I:FD<=IFD

  $K GIF989CP IFCC<; F== K?< ?L>< DFLEK8@E K?8K KFN<IJ FM<I FLI M@CC8><, 2?<I<;@;K?<DFLEK8@E:FD<=IFD

  /?< J8D< GC8:< 8J 8CC JKFE< @K @J K?< 9FE<J F= 4D@I K?< GI@DFIe;@8C>@8EK, 2?<I<;@;K?<GI@DFI;@8C>@8EK4D@I:FD<=IFD

  !IFDK?<>I<8K89PJJ"@EELE>8>8G, 2?<I<;@;K?<>I<8K89PJJ"@EELE>8>8G:FD<=IFD

  )<M<I8JBK?8KHL<JK@FE

FEJ@;<I K?< J<<D@E> G8I8;FO F= K?< !@IJK 8LJ< .:@<E:< ?8JKI8:<; <M<EKJ 98:B KF K?< @> 8E> 9LK N?P ;@; K?< @> 8E> ?8GeG<E $KcJ 8CC N<CC 8E; >FF; KF J8P K?8K K?< Q<IF F= K@D< 9<>@EJ 8K K?<@> 8E>gK?8K K?<I< @J EFK?@E> 9<=FI< K?< @> 8E> @E K?< FI;@E8IP =CFN F= D@ELK<J 8E; ?FLIJ LK J8P@E> K?@J GI<JLD<J FLI G?PJ@:8C

C8N N?@:? @KJ<C= 8GG<8IJ ?@>?CP JKIL:KLI<; @K :8CCJ FLK =FI <OGC8E8eK@FE 2?<I< ;@; K?< G?PJ@:8C C8NJ :FD< =IFD 4FL :FLC; J8P K?8K N<cI< 8CC 8 :FDGLK<I J@DLC8K@FE 9LK K?<E K?< :FDGLK<I J@DLC8K@FE@J ILEE@E> FE JFD< FK?<I NFIC;cJ C8NJ F= G?PJ@:JgN?<I< ;@; thoseC8NJF=G?PJ@:J:FD<=IFD

 KK?@JGF@EKJFD<G<FGC<J8P`"F;a

2?8K :FLC; GFJJ@9CP D8B< 8EPFE< <M<E 8 ?@>?CP I<C@>@FLJ G<IeJFE K?@EB K?@J <M<E helped 8EJN<I K?< G8I8;FO F= K?< !@IJK 8LJ<2?P NFLC;EcK PFL 8LKFD8K@:8CCP 8JB `2?<I< ;@; "F; :FD<=IFDa .8P@E> `"F; @J LE:8LJ<;a FI `"F; :I<8K<; #@DJ<C=a C<8M<JLJ @E <O8:KCP K?< J8D< GFJ@K@FE 8J `/@D< 9<>8E N@K? K?< @> 8E>a2< ALJK 8JB N?P K?< N?FC< D<K8JPJK<D <O@JKJ @E K?< =@IJK GC8:< FI

 N?PJFD<<M<EKJ9LKEFKFK?<IJ8I<8CCFN<;KF9<LE:8LJ<;

(P GLIGFJ< ?<I< @J EFK KF ;@J:LJJ K?< J<<D@E> G8I8;FO F= K?<!@IJK 8LJ< 9LK KF 8JB N?P 8EPFE< NFLC; K?@EB `"F;a could I<JFCM<K?< G8I8;FO .8P@E> `"F;a @J 8 N8P F= 9<CFE>@E> KF 8 KI@9< N?@:?

http://lesswrong.com/lw/it/semantic_stopsigns/  +8><b+IF=<JJ@E>8E;?<<I@E>c

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 152/309

 >@M<J G<FGC< 8 DFK@M< KF J8P @K 8J F=K<E 8J GFJJ@9C<gJFD< G<FGC<<M<E J8P @K =FI HL<JK@FEJ C@B< `2?P ;@; K?@J ?LII@:8E< JKI@B< )<N *IC<8EJa M<E JF PFLc; ?FG< G<FGC< NFLC; EFK@:< K?8K FE K?<

 particular GLQQC< F= K?< !@IJK 8LJ< J8P@E> `"F;a ;F<JEcK ?<CG $K;F<JEcK D8B< K?< G8I8;FO J<<D 8EP C<JJ G8I8;FO@:8C even if true.#FN:FLC;8EPFE< not EFK@:<K?@J

 %FE8K?8E 28CC8:< JL>><JK<; K?8K `"F;a =LE:K@FEJ 8J 8 semantic stopsign gK?8K @K @JEcK 8 GIFGFJ@K@FE8C 8JJ<IK@FE JF DL:? 8J 8 :F>eE@K@M< KI8==@: J@>E8C ;F EFK K?@EB G8JK K?@J GF@EK .8P@E> `"F;a;F<JEcK JF DL:? I<JFCM< K?< G8I8;FO 8J GLK LG 8 :F>E@K@M< KI8==@:J@>E8C KF ?8CK K?< F9M@FLJ :FEK@EL8K@FE F= K?< HL<JK@FEe8E;e8EJN<I:?8@E

*= :FLIJ< you’d E<M<I ;F K?8K 9<@E> 8 >FF; 8E; GIFG<I 8K?<@JKI@>?K LK `"F;a @JEcK K?< only J<D8EK@: JKFGJ@>E ALJK K?< F9M@FLJ=@IJK<O8DGC<

/?< KI8EJ?LD8E K<:?EFCF>@<JgDFC<:LC8I E8EFK<:?EFCF>P 8;e M8E:<; 9@FK<:? ><E<K<:? IK@=@:@8C $EK<CC@><E:< <K :<K<I8gGFJ<KFL>? GFC@:P HL<JK@FEJ 2?8K B@E; F= IFC< @= 8EP J?FLC; 8 >FMe<IED<EK K8B< @E JLG<IM@J@E> 8 G8I<EKcJ :?F@:< F= ><E<J =FI K?<@I:?@C; FLC; G8I<EKJ ;<C@9<I8K<CP :?FFJ< ><E<J =FI J:?@QFG?I<E@8

$= <E?8E:@E> 8 :?@C;cJ @EK<CC@><E:< @J <OG<EJ@M< J?FLC; >FM<IED<EKJ?<CG <EJLI< 8::<JJ KF GI<M<EK K?< <D<I><E:< F= 8 :F>E@K@M< <C@K<4FL :8E GIFGFJ< M8I@FLJ @EJK@KLK@FEJ KF 8EJN<I K?<J< GFC@:P HL<JeK@FEJg=FI <O8DGC< K?8K GI@M8K< :?8I@K@<J J?FLC; GIFM@;< =@E8E:@8C8@; =FI @EK<CC@><E:< <E?8E:<D<EKg9LK K?< F9M@FLJ E<OK HL<JK@FE @J`2@CC K?@J @EJK@KLK@FE 9< <==<:K@M<a $= N< I<CP FE GIF;L:K C@89@C@KP C8NJL@KJ KF GI<M<EK :FIGFI8K@FEJ =IFD 9L@C;@E> ?8ID=LC E8EFK<:?

 N@CCK?8KI<8CCP work? 

$ BEFN JFD<FE< N?FJ< 8EJN<I KF <M<IP FE< F= K?<J< HL<JK@FEJ

@J `'@9<I8C ;<DF:I8:Pa /?8KcJ @K /?8KcJ ?@J 8EJN<I $= PFL 8JB K?<F9M@FLJ HL<JK@FE F= `#FN N<CC ?8M< C@9<I8C ;<DF:I8:@<J G<I=FID<;?@JKFI@:8CCP FE GIF9C<DJ K?@J KI@:BPa FI `2?8K @= C@9<I8C ;<DF:I8e:P ;F<J JFD<K?@E> JKLG@;a K?<E PFLcI< 8E 8LKF:I8K FI C@9<IKFG@8EFI FK?<IN@J< 8 M<IP M<IP 98; G<IJFE )F FE< @J 8CCFN<; KF HL<JK@FE;<DF:I8:P

$ FE:< :8CC<; K?@J B@E; F= K?@EB@E> `K?< ;@M@E< I@>?K F= ;<DF:eI8:Pa LK @K @J DFI< GI<:@J< KF J8P K?8K `<DF:I8:Pa =LE:K@FE<;=FI ?@D 8J 8 J<D8EK@: JKFGJ@>E $= 8EPFE< ?8; J8@; KF ?@D `/LIE @K

%1+,*!'-+ &+/*+,'%1+,*!'-+)-+,!'&+

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 153/309

FM<I KF K?< F:8eFC8 :FIGFI8K@FEa ?< NFLC; ?8M< 8JB<; K?< F9M@eFLJ E<OK HL<JK@FEJ `2?P 2?8K N@CC K?< F:8eFC8 :FIGFI8K@FE;F 89FLK @K 2?P J?FLC; N< KILJK K?<D #8M< K?<P ;FE< N<CC @E

K?<G8JKFE<HL8CCPKI@:BPGIF9C<DJa*I JLGGFJ< K?8K JFD<FE< J8PJ `(<O@:8EeD<I@:8EJ 8I< GCFKK@E> 

KF I<DFM< 8CC K?< FOP><E @E 8IK?cJ 8KDFJG?<I<a 4FLc; GIF989CP 8JB `2?P NFLC; K?<P ;F that?  FEcK (<O@:8EeD<I@:8EJ ?8M< KF9I<8K?< KFF F (<O@:8EeD<I@:8EJ <M<E =LE:K@FE 8J 8 LE@=@<;:FEJG@I8:Pa $= PFL ;FEcK 8JB K?<J< F9M@FLJ E<OK HL<JK@FEJ N?<EJFD<FE< J8PJ `FIGFI8K@FEJ 8I< GCFKK@E> KF I<DFM< 8IK?cJ FOPe

 ><Ea K?<E `FIGFI8K@FEJa =LE:K@FEJ =FI PFL 8J 8 J<D8EK@: JKFGJ@>E

< :8I<=LC ?<I< EFK KF :I<8K< 8 E<N ><E<I@: :FLEK<I8I>LD<EK

8>8@EJK K?@E>J PFL ;FEcK C@B<ga*? @KcJ ALJK 8 JKFGJ@>Ea )F NFI; @J8 JKFGJ@>E F= @KJ<C= K?< HL<JK@FE @J N?<K?<I 8 NFI; ?8J K?8K <==<:KFE 8 G8IK@:LC8I G<IJFE #8M@E> JKIFE> <DFK@FEJ 89FLK JFD<K?@E> ;F<JEcK HL8C@=P @K 8J 8 JKFGJ@>E $cD EFK <O8:KCP =FE; F= K<IIFI@JKJFI =<8I=LC F= GI@M8K< GIFG<IKP K?8K ;F<JEcK D<8E `/<IIFI@JKJa FI`8G@K8C@JDa 8I< :F>E@K@M< KI8==@: J@>E8CJ LEKF D< h/?< NFI; `@EeK<CC@><E:<a ;@; FE:< ?8M< K?8K <==<:K FE D< K?FL>? EF CFE><Ii2?8K ;@JK@E>L@J?<J 8 J<D8EK@: JKFGJ@>E @J failure to consider the obvi-

ous next question.

+8><b!<<C@E>-8K@FE8Cc

+%&,!+,'(+!&+

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 154/309

18. Mysterious Answers to MysteriousQuestions

$D8>@E< CFFB@E> 8K PFLI ?8E; 8E; BEFN@E> EFK?@E> F= :<CCJ EFK?e@E> F= 9@F:?<D@JKIP EFK?@E> F= ) 4FLcM< C<8IE<; JFD< 8E8KFeDP =IFD ;@JJ<:K@FE JF PFL BEFN PFLI ?8E; :FEK8@EJ DLJ:C<J 9LK

 PFL ;FEcK BEFN N?P DLJ:C<J DFM< @EJK<8; F= CP@E> K?<I< C@B< :C8P4FLI ?8E; @J ALJK_ JKL==_ 8E; =FI JFD< I<8JFE @K DFM<J LE;<I PFLI;@I<:K@FE$JK?@JEFKD8>@:

`/?<8E@D8C9F;P;F<JEFK8:K8J8K?<IDF;PE8D@:<E>@E<_:FEJ:@FLJE<JJK<8:?<J<M<IP@E;@M@;L8CK?8KK?<P8I<KFJFD<<OK<EKJL9A<:KKFK?<;@I<:K@FEF=?@J

 N@CC$K8GG<8IJK?<I<=FI<K?8K8E@D8K<;:I<8KLI<J?8M<K?<GFN<IF=@DD<;@8K<CP8GGCP@E>KF:<IK8@EDFM@E> G8IK@:C<JF=D8KK<IN@K?@EK?<@I9F;@<J=FI:<J9PN?@:?K?<DFK@FEJF=K?<J<G8IK@:C<J8I<;@I<:K<;KFGIF;L:<;<I@M<;D<:?8E@:8C<==<:KJ_/?<@E=CL<E:<F=8E@D8CFI

 M<><K89C<C@=<FED8KK<I@J@E=@E@K<CP9<PFE;K?<I8E><F= 8EPJ:@<EK@=@:@EHL@IP?@K?<IKF<EK<I<;FE$KJGFN<IF= 

;@I<:K@E>K?<DFK@FEJF=DFM@E>G8IK@:C<J@EK?<;<DFEJKI8K<;;8@CPD@I8:C<F=FLI?LD8E=I<<eN@CC8E;@EK?<>IFNK?F=><E<I8K@FE8=K<I><E<I8K@FEF=GC8EKJ=IFD8J@E>C<J<<;8I<@E=@E@K<CP;@==<I<EK=IFD8EPGFJJ@9C<I<JLCKF=K?<=FIKL@KFLJ:FE:LII<E:<F=8KFDJ_(F;<IE9@FCF>@JKJN<I<:FD@E>FE:<DFI<KFK?<8::<GK8E:<F= JFD<K?@E>8E;K?8KN8J8M@K8CGI@E:@GC<a

 g'FI;&<CM@E

/?@J N8J K?< K?<FIP F=  vitalism K?8K K?< DPJK<I@FLJ ;@==<I<E:<9<KN<<E C@M@E> D8KK<I 8E; EFEeC@M@E> D8KK<I N8J <OGC8@E<; 9P 8Eelan vital FI vis vitalis C8E M@K8C @E=LJ<; C@M@E> D8KK<I 8E; :8LJ<; @KKF DFM< 8J :FEJ:@FLJCP ;@I<:K<; C8E M@K8C G8IK@:@G8K<; @E :?<D@:8CKI8EJ=FID8K@FEJ N?@:? EF D<I< EFEeC@M@E> G8IK@:C<J :FLC; LE;<Ie

 >Fg2Y?C<IcJ C8K<I JPEK?<J@J F= LI<8 8 :FDGFE<EK F= LI@E< N8J8 D8AFI 9CFN KF K?< M@K8C@JK@: K?<FIP 9<:8LJ< @K J?FN<; K?8K D<I<chemistry :FLC;;LGC@:8K<8GIF;L:KF=9@FCF>P

http://lesswrong.com/lw/iu/mysterious_answers_to_mysterious_questions/ 

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 155/309

8CC@E> `<C8E M@K8Ca 8E <OGC8E8K@FE <M<E 8 =8B< <OGC8E8K@FE C@B<G?CF>@JKFE @J GIF989CP >@M@E> @K KFF DL:? :I<;@K $K =LE:K@FE<;GI@D8I@CP 8J 8 :LI@FJ@KPeJKFGG<I 4FL J8@; `2?Pa 8E; K?< 8EJN<I

 N8J`C8EM@K8Ca2?<E PFL J8P `C8E M@K8Ca @K feels C@B< PFL BEFN N?P PFLI ?8E;

DFM<J 4FL ?8M< 8 C@KKC< :8LJ8C ;@8>I8D @E PFLI ?<8; K?8K J8PJj`C8E M@K8Cak e j?8E; DFM<Jk LK 8:KL8CCP PFL BEFN EFK?@E> PFL;@;EcK BEFN 9<=FI< 4FL ;FEcK BEFN J8P N?<K?<I PFLI ?8E; N@CC

 ><E<I8K< ?<8K FI 89JFI9 ?<8K LEC<JJ PFL ?8M< F9J<IM<; K?< =8:K8CI<8;P @= EFK PFL NFEcK 9< 89C< KF GI<;@:K @K @E 8;M8E:< 4FLI :LeI@FJ@KP =<<CJ J8K<; 9LK @K ?8JEcK 9<<E =<; .@E:< PFL :8E J8P `2?PC8E M@K8Ca KF 8EP GFJJ@9C< F9J<IM8K@FE @K @J <HL8CCP >FF; 8K <OeGC8@E@E> 8CC FLK:FD<J 8 ;@J>L@J<; ?PGFK?<J@J F= D8O@DLD <EKIFGP<K:<K<I8

LK K?< >I<8K<I C<JJFE C@<J @E K?< M@K8C@JKJc I<M<I<E:< =FI K?< <C8E M@K8C K?<@I <8><IE<JJ KF GIFEFLE:< @K 8 DPJK<IP 9<PFE; 8CC J:@<E:<(<<K@E> K?< >I<8K ;I8>FE 0EBEFNE K?< M@K8C@JKJ ;@; EFK ;I8N K?<@IJNFI;J KF ;F 98KKC< 9LK 9FN<; K?<@I E<:BJ @E JL9D@JJ@FE /?<P KFFB GI@;< @E K?<@I @>EFI8E:< D8;< 9@FCF>P @EKF 8  sacred DPJK<IP8E; K?<I<9P 9<:8D< CF8K? KF I<C@EHL@J? K?<@I @>EFI8E:<  N?<E <M@e

;<E:<:8D<BEF:B@E>/?< .<:I<K F= '@=< N8J infinitely beyond the reach of science! )FK ALJK

8 little 9<PFE; D@E; PFL 9LK infinitely 9<PFE; 'FI; &<CM@E JLI< ;@; ><K8KI<D<E;FLJ<DFK@FE8CB@:BFLKF= not knowing something.

LK @>EFI8E:< <O@JKJ @E K?< D8G EFK @E K?< K<II@KFIP $= $ 8D @>eEFI8EK 89FLK 8 G?<EFD<EFE K?8K @J 8 =8:K 89FLK DP FNE JK8K< F= D@E; EFK 8 =8:K 89FLK K?< G?<EFD<EFE @KJ<C= G?<EFD<EFE :8E seem DPJK<I@FLJ KF JFD< G8IK@:LC8I G<IJFE /?<I< 8I< EF G?<EFD<e

E8 N?@:? 8I< DPJK<I@FLJ F= K?<DJ<CM<J /F NFIJ?@G 8 G?<EFD<EFE9<:8LJ< @K J<<DJ JF NFE;<I=LCCP DPJK<I@FLJ @J KF NFIJ?@G PFLI FNE@>EFI8E:<

1@K8C@JD J?8I<; N@K? G?CF>@JKFE K?< <IIFI F=  encapsulating the mystery as a substance. !@I< N8J DPJK<I@FLJ 8E; K?< G?CF>@JKFE K?<FIP 

+8><b!8B<OGC8E8K@FEJc +8><b!8B<8LJ8C@KPc +8><b.<D8EK@:.KFGJ@>EJc +8><b!8B<8LJ8C@KPc

http://lesswrong.com/lw/h8/tsuyoku_naritai_i_want_to_become_stronger/  http://yudkowsky.net/virtues/ 

%1+,*!'-+ &+/*+,'%1+,*!'-+)-+,!'&+

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 156/309

<E:8GJLC8K<; K?< DPJK<IP @E 8 DPJK<I@FLJ JL9JK8E:< :8CC<; `G?CFe >@JKFEa '@=< N8J 8 J8:I<; DPJK<IP 8E; M@K8C@JD <E:8GJLC8K<; K?<J8:I<; DPJK<IP @E 8 DPJK<I@FLJ JL9JK8E:< :8CC<; `<C8E M@K8Ca )<@K?<I

8EJN<I ?<CG<; :FE:<EKI8K< K?< DF;<CcJ GIF989@C@KP ;<EJ@KP  gD8B<JFD< FLK:FD<J <8J@<I KF <OGC8@E K?8E FK?<IJ /?< `<OGC8E8K@FEa ALJK NI8GG<;LGK?<HL<JK@FE8J8JD8CC?8I;FG8HL<9C8:B98CC

$E 8 :FD<;P NI@KK<E 9P (FC@<I< 8 G?PJ@:@8E <OGC8@EJ K?< GFN<IF= 8 JFGFI@=@: 9P J8P@E> K?8K @K :FEK8@EJ 8 `;FID@K@M< GFK<E:Pa.8D< GI@E:@GC< $K @J 8 =8@CLI< F= ?LD8E GJP:?FCF>P K?8K =8:<; N@K?8 DPJK<I@FLJ G?<EFD<EFE N< DFI< I<8;@CP GFJKLC8K< DPJK<I@FLJ@E?<I<EKJL9JK8E:<JK?8E:FDGC<OLE;<ICP@E>GIF:<JJ<J

LK K?< ;<<G<I =8@CLI< @J JLGGFJ@E> K?8K 8E answer :8E 9< DPJeK<I@FLJ $= 8 G?<EFD<EFE =<<CJ DPJK<I@FLJ K?8K @J 8 =8:K 89FLK FLIJK8K< F= BEFNC<;>< EFK 8 =8:K 89FLK K?< G?<EFD<EFE @KJ<C= /?<

 M@K8C@JKJ J8N 8 DPJK<I@FLJ >8G @E K?<@I BEFNC<;>< 8E; GFJKLC8K<; 8 DPJK<I@FLJ JKL== K?8K GCL>><; K?< >8G $E ;F@E> JF K?<P D@O<; LGK?< D8G N@K? K?< K<II@KFIP CC :FE=LJ@FE 8E; 9<N@C;<ID<EK <O@JK @EK?<D@E;EFK@E<E:8GJLC8K<;JL9JK8E:<J

/?@J @J K?< LCK@D8K< 8E; =LCCP ><E<I8C <OGC8E8K@FE =FI N?P 8>8@E8E; 8>8@E @E ?LD8E@KPcJ ?@JKFIP G<FGC< 8I< J?F:B<; KF ;@J:FM<I K?8K

8E @E:I<;@9CP DPJK<I@FLJ HL<JK@FE ?8J 8 EFEeDPJK<I@FLJ 8EJN<I(PJK<IP@J8GIFG<IKPF=HL<JK@FEJEFK8EJN<IJ

/?<I<=FI< $ :8CC K?<FI@<J JL:? 8J M@K8C@JD mysterious answers to mysterious questions

/?<J< 8I< K?< J@>EJ F= DPJK<I@FLJ 8EJN<IJ KF DPJK<I@FLJ HL<JeK@FEJ

[ !@IJKK?<<OGC8E8K@FE8:KJ8J8 :LI@FJ@KPeJKFGG<I I8K?<IK?8E8E 8EK@:@G8K@FEe:FEKIFCC<I

[ .<:FE;K?<?PGFK?<J@J?8JEFDFM@E>G8IKJgK?<DF;<C@JEFK8JG<:@=@::FDGC<OD<:?8E@JD9LK89C8EBCPJFC@;JL9JK8E:<FI=FI:</?<DPJK<I@FLJJL9JK8E:<FIDPJK<I@FLJ=FI:<D8P9<J8@;KF9<?<I<FIK?<I<KF :8LJ<K?@JFIK?8K9LKK?<I<8JFEN?PK?<DPJK<I@FLJ=FI:<9<?8M<JK?LJ@JNI8GG<;@E89C8EBLE@KP

+8><b!F:LJ4FLI0E:<IK8@EKPc +8><b.<D8EK@:.KFGJ@>EJc

+8><b(8B@E><C@<=J+8P-<EKh@EEK@:@G8K<;OG<I@<E:<Jic +8><b!8B<8LJ8C@KPc

%1+,*!'-+ &+/*+,'%1+,*!'-+)-+,!'&+

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 157/309

[ /?@I;K?FJ<N?FGIF==<IK?<<OGC8E8K@FE :?<I@J?K?<@I@>EFI8E:<K?<PJG<8BGIFL;CPF=?FNK?<G?<EFD<EFE;<=<8KJFI;@E8IPJ:@<E:<FI@JLEC@B<D<I<CPDLE;8E<

G?<EFD<E8[ !FLIK? even after the answer is given, the phenomenon is still a mystery 8E;GFJJ<JJ<JK?<J8D<HL8C@KPF=NFE;<I=LC@E<OGC@:89@C@KPK?8K@K?8;8KK?<JK8IK

http://yudkowsky.net/virtues/ 

%1+,*!'-+ &+/*+,'%1+,*!'-+)-+,!'&+

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 158/309

19. The Futility of Emergence

Prerequisites: <C@<= @E <C@<=  !8B< OGC8E8K@FEJ !8B< 8LJ8Ce

@KP  (PJK<I@FLJEJN<IJKF(PJK<I@FLJ,L<JK@FEJ

/?< =8@CLI<J F= G?CF>@JKFE 8E; M@K8C@JD 8I< ?@JKFI@:8C ?@E;eJ@>?K 8I< $ JK<G FLK FE 8 C@D9 8E; E8D< JFD< current  K?<FIP 

 N?@:?$;<<D8E8CF>FLJCP=C8N<;

$ E8D< emergence FI emergent phenomena gLJL8CCP ;<=@E<; 8J K?<JKL;P F= JPJK<DJ N?FJ< ?@>?eC<M<C 9<?8M@FIJ 8I@J< FI `<D<I><a =IFDK?< @EK<I8:K@FE F= D8EP CFNeC<M<C <C<D<EKJ h2@B@G<;@8  `/?<

 N8P :FDGC<O JPJK<DJ 8E; G8KK<IEJ 8I@J< FLK F= 8 DLCK@GC@:@KP F= I<Ce

8K@M<CP J@DGC< @EK<I8:K@FEJai /8B<E C@K<I8CCP K?8K ;<J:I@GK@FE =@KJ<M<IP G?<EFD<EFE @E FLI LE@M<IJ< 89FM< K?< C<M<C F= @E;@M@;L8CHL8IBJ N?@:? @J G8IK F= K?< GIF9C<D $D8>@E< GF@EK@E> KF 8 D8IB<K:I8J? 8E; J8P@E> `$KcJ EFK 8 HL8IBa F<J K?8K =<<C C@B< 8E <OGC8eE8K@FE )F /?<E E<@K?<I J?FLC; J8P@E> `$KcJ 8E <D<I><EK G?<eEFD<EFEa

$KcJ K?< EFLE `<D<I><E:<a K?8K $ GIFK<JK I8K?<I K?8E K?< M<I9`<D<I><J =IFDa /?<I<cJ EFK?@E> NIFE> N@K? J8P@E> `3 <D<I><J=IFD 4a N?<I< 4 @J JFD< JG<:@=@: ;<K8@C<; DF;<C N@K? @EK<IE8CDFM@E> G8IKJ `I@J<J =IFDa @J 8EFK?<I C<>@K@D8K< G?I8J< K?8KD<8EJ <O8:KCP K?< J8D< K?@E> "I8M@KP 8I@J<J =IFD K?< :LIM8KLI< F= JG8:<K@D< 8::FI;@E> KF K?< JG<:@=@: D8K?<D8K@:8C DF;<C F= "<Ee<I8C -<C8K@M@KP ?<D@JKIP 8I@J<J =IFD @EK<I8:K@FEJ 9<KN<<E 8KFDJ8::FI;@E>KFK?<JG<:@=@:DF;<CF=HL8EKLD<C<:KIF;PE8D@:J

)FN JLGGFJ< $ J?FLC; J8P K?8K >I8M@KP @J <OGC8@E<; 9P `8I@J<E:<aFI K?8K :?<D@JKIP @J 8E `8I@J@E> G?<EFD<EFEa 8E; :C8@D K?8K 8J DP <OGC8E8K@FE

http://lesswrong.com/lw/iv/the_futility_of_emergence/  +8><b<C@<=@E<C@<=c +8><b!8B<OGC8E8K@FEJc +8><b!8B<8LJ8C@KPc +8><b(PJK<I@FLJEJN<IJKF(PJK<I@FLJ,L<JK@FEJc +8><b!8B<8LJ8C@KPc +8><b(PJK<I@FLJEJN<IJKF(PJK<I@FLJ,L<JK@FEJc +8><b#@E;J@>?K<M8CL<J.:@<E:<c

+8><b#@E;J@>?K9@8Jc http://en.wikipedia.org/wiki/Emergence

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 159/309

/?< G?I8J< `<D<I><J =IFDa @J 8::<GK89C< ALJK C@B< `8I@J<J =IFDaFI `@J :8LJ<; 9Pa 8I< 8::<GK89C< @= K?< G?I8J< GI<:<;<J JFD< JG<:@=e@:DF;<CKF9<AL;><;FE@KJFNED<I@KJ

#FN<M<I K?@J @J not  K?< N8P `<D<I><E:<a @J :FDDFECP LJ<;`D<I><E:<a@J:FDDFECPLJ<;8J8E<OGC8E8K@FE@E@KJFNEI@>?K

$ ?8M< CFJK KI8:B F= ?FN D8EP K@D<J $ ?8M< ?<8I; G<FGC< J8P`$EK<CC@><E:< @J 8E <D<I><EK G?<EFD<EFEa 8J @= K?8K <OGC8@E<; @EeK<CC@><E:< /?@J LJ8>< =@KJ 8CC K?< :?<:BC@JK @K<DJ =FI 8 DPJK<I@FLJ8EJN<I KF 8 DPJK<I@FLJ HL<JK@FE 2?8K ;F PFL BEFN 8=K<I PFL?8M< J8@; K?8K @EK<CC@><E:< @J `<D<I><EKa 4FL :8E D8B< EF E<N GI<;@:K@FEJ 4FL ;F EFK BEFN 8EPK?@E> 89FLK K?< 9<?8M@FI F= I<8Ce

 NFIC; D@E;J K?8K PFL ;@; EFK BEFN 9<=FI< $K =<<CJ C@B< PFL 9<C@<M<8 E<N =8:K 9LK PFL ;FEcK 8EK@:@G8K< 8EP ;@==<I<EK FLK:FD<J 4FLI:LI@FJ@KP =<<CJ J8K<; 9LK @K ?8J EFK 9<<E =<; /?< ?PGFK?<J@J ?8JEF DFM@E> G8IKJgK?<I<cJ EF ;<K8@C<; @EK<IE8C DF;<C KF D8E@GLC8K</?FJ< N?F GIF==<I K?< ?PGFK?<J@J F= `<D<I><E:<a :FE=<JJ K?<@I @>eEFI8E:< F= K?< @EK<IE8CJ 8E; K8B< GI@;< @E @K K?<P :FEKI8JK K?<J:@<E:<F=`<D<I><E:<aKFFK?<IJ:@<E:<JD<I<CPDLE;8E<

 E; <M<E 8=K<I K?< 8EJN<I F= `2?P D<I><E:<a @J >@M<E the phenomenon is still a mystery 8E; GFJJ<JJ<J K?< J8D< J8:I<; @DG<E<e

KI89@C@KP@K?8;8KK?<JK8IK  =LE <O<I:@J< @J KF <C@D@E8K< K?< 8;A<:K@M< `<D<I><EKa =IFD 8EP 

J<EK<E:< @E N?@:? @K 8GG<8IJ 8E; J<< @= K?< J<EK<E:< J8PJ 8EPK?@E> ;@==<I<EK

[ Before: #LD8E@EK<CC@><E:<@J8E<D<I><EKGIF;L:KF= E<LIFEJ=@I@E>

[ After: #LD8E@EK<CC@><E:<@J8GIF;L:KF=E<LIFEJ=@I@E>[ Before: /?<9<?8M@FIF=K?<8EK:FCFEP@JK?<<D<I><EK

FLK:FD<F=K?<@EK<I8:K@FEJF=D8EP@E;@M@;L8C8EKJ

[ After: /?<9<?8M@FIF=K?<8EK:FCFEP@JK?<FLK:FD<F= K?<@EK<I8:K@FEJF=D8EP@E;@M@;L8C8EKJ

[ Even better: :FCFEP@JD8;<F=8EKJ2<:8EJL::<JJ=LCCP GI<;@:KJFD<8JG<:KJF=:FCFEP9<?8M@FILJ@E>DF;<CJK?8K@E:CL;<FECP@E;@M@;L8C8EKJN@K?FLK8EP>CF98C:FCFEP 

 M8I@89C<JJ?FN@E>K?8KN<LE;<IJK8E;?FNK?FJ<:FCFEP 9<?8M@FIJ8I@J<=IFD8EK9<?8M@FIJ

+8><b(PJK<I@FLJEJN<IJKF(PJK<I@FLJ,L<JK@FEJc

,-,!$!,1'%*&

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 160/309

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 161/309

20. Say Not “Complexity”

*E:<LGFE8K@D<_

/?@J @J 8 JKFIP =IFD N?<E $ =@IJK D<K (8I:<CCF N@K? N?FD $ NFLC; C8K<I NFIB =FI 8 P<8I FE $ K?<FIP 9LK 8K K?@J GF@EK $ ?8; EFK P<K 8::<GK<; ?@D 8J DP 8GGI<EK@:< $ BE<N K?8K ?< :FDG<K<; 8KK?< E8K@FE8C C<M<C @E D8K?<D8K@:8C 8E; :FDGLK@E> FCPDG@8;J N?@:?JL==@:<; KF 8KKI8:K DP 8KK<EK@FE =FI 8 :CFJ<I CFFB 9LK $ ;@;EcK BEFN 

 P<K@=?<:FLC;C<8IEKFK?@EB89FLK$

$ ?8; 8JB<; (8I:<CCF KF J8P ?FN ?< K?FL>?K 8E $ D@>?K ;@Je:FM<I ?FN KF JFCM< 8 -L9@BcJ L9< )FK @E 8 GI<GIF>I8DD<; N8P

 N?@:? @J KI@M@8C 9LK I8K?<I ?FN K?< $ @KJ<C= D@>?K =@>LI< FLK K?<C8NJ F= K?< -L9@B LE@M<IJ< 8E; I<8JFE FLK ?FN KF <OGCF@K K?<D#FN NFLC; 8E $ invent for itself  K?< :FE:<GK F= 8E `FG<I8KFIa FI`D8:IFaN?@:?@JK?<B<PKFJFCM@E>K?<-L9@BcJL9<

 K JFD< GF@EK @E K?@J ;@J:LJJ@FE (8I:<CCF J8@; `2<CC $ K?@EBK?<$E<<;J:FDGC<O@KPKF;F38E;:FDGC<O@KPKF;F4ga

 E;$J8@;`FEcKJ8Pbcomplexity’.a

(8I:<CCFJ8@;`2?PEFKa

$ J8@; `FDGC<O@KP J?FLC; E<M<I 9< 8 >F8C @E @KJ<C= 4FL D8P E<<; KF LJ< 8 G8IK@:LC8I 8C>FI@K?D K?8K 8;;J JFD< 8DFLEK F= :FDeGC<O@KP 9LK :FDGC<O@KP =FI K?< J8B< F= :FDGC<O@KP ALJK D8B<J K?@E>J?8I;<Ia h$ N8J K?@EB@E> F= 8CC K?< G<FGC< N?FD $ ?8; ?<8I; 8;MFe:8K@E> K?8K K?< $EK<IE<K NFLC; `N8B< LGa 8E; 9<:FD< 8E $ N?<E@K9<:8D<`JL==@:@<EKCP:FDGC<Oai

 E; (8I:<CCF J8@; `LK K?<I<cJ >FK KF 9< some 8DFLEK F= :FDeGC<O@KPK?8K;F<J@Ka

$ :CFJ<; DP <P<J 9I@<=CP 8E; KI@<; KF K?@EB F= ?FN KF <OGC8@E @K

8CC @E NFI;J /F D< J8P@E> b:FDGC<O@KPc J@DGCP felt  C@B< K?< NIFE> DFM< @E K?< $ ;8E:< )F FE< :8E K?@EB =8JK <EFL>? KF ;<C@9<I8K<@E NFI;J 89FLK <8:? J<EK<E:< F= K?<@I JKI<8D F= :FEJ:@FLJE<JJ =FIK?8K NFLC; I<HL@I< 8E @E=@E@K< I<:LIJ@FE 2< K?@EB @E NFI;J 9LKFLI JKI<8D F= :FEJ:@FLJE<JJ @J JK<<I<; 9<CFN K?< C<M<C F= NFI;J 9P K?<KI8@E<;e@EI<DE8EKJF=G8JK@EJ@>?KJ8E;?8IJ?<OG<I@<E:<_

$ J8@; `@; PFL I<8; /<:?E@:8C OGC8E8K@FE F= /<:?E@:8C OeGC8E8K@FEa

http://lesswrong.com/lw/ix/say_not_complexity/  http://yudkowsky.net/bayes/technical.html 

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 162/309

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 163/309

$ JLJG<:K K?8K @E 8:8;<D@8 K?<I< @J 8 ?L>< GI<JJLI< KF JN<<GGIF9C<DJ LE;<I K?< IL> JF K?8K PFL :8E GI<J<EK 8 G8G<I N@K? K?< 8GeG<8I8E:< F= :FDGC<K<E<JJ 4FLcCC ><K DFI< BL;FJ =FI 8 J<<D@E>CP 

:FDGC<K< DF;<C K?8K @E:CL;<J JFD< `<D<I><EK G?<EFD<E8 ` M<IJLJ8E <OGC@:@KCP @E:FDGC<K< D8G N?<I< K?< C89<C J8PJ `$ >FK EF :CL< ?FN K?@J G8IK NFIBJa FI `K?<E 8 D@I8:C< F::LIJa AFLIE8C D8P EFK <M<E8::<GK K?< C8KK<I G8G<I J@E:< N?F BEFNJ 9LK K?8K K?< LEBEFNEJK<GJ 8I< I<8CCP N?<I< <M<IPK?@E> @EK<I<JK@E> ?8GG<EJ E; P<J @KJFD<K@D<J ?8GG<EJ K?8K 8CC K?< EFEeD8>@:8C G8IKJ F= PFLI D8G KLIEFLK KF 8CJF 9< EFEe@DGFIK8EK /?8KcJ K?< GI@:< PFL JFD<K@D<J G8P=FI <EK<I@E> @EKF K<II8 @E:F>E@K8 8E; KIP@E> KF JFCM< GIF9C<DJ incre- mentally. LK K?8K D8B<J @K <M<E more @DGFIK8EK KF know N?<E PFL

8I<EcK =@E@J?<; P<K (FJKCP G<FGC< ;FEcK ;8I< KF <EK<I K<II8 @E:F>eE@K88K8CC=FIK?<;<8;CP=<8IF=N8JK@E>K?<@IK@D<

 E; @= PFLcI< NFIB@E> FE 8 I<MFCLK@FE8IP $ JK8IKLG K?<I< @J 8E<M<E ?L><I GI<JJLI< KF JN<<G GIF9C<DJ LE;<I K?< IL> FI PFL N@CC?8M< KF 8;D@K KF PFLIJ<C=  K?8K PFL ;FEcK BEFN ?FN KF 9L@C; 8E $

 P<K 8E; PFLI :LII<EK C@=<eGC8EJ N@CC :FD< :I8J?@E> ;FNE @E IL@EJ8IFLE; PFLI <8IJ LK G<I?8GJ $ 8D FM<Ie<OGC8@E@E>  J@E:< JB@GeFM<I ?8GG<EJ 9P ;<=8LCK @E ?LD8EJ @= PFLcI< CFFB@E> =FI <O8DGC<J

ALJK N8K:? G<FGC< ;@J:LJJ@E> I<C@>@FE FI G?@CFJFG?P FI JG@I@KL8C@KP FI8EPJ:@<E:<@EN?@:?K?<PN<I<EFKGIF=<JJ@FE8CCPKI8@E<;

(8I:<CCF 8E; $ ;<M<CFG<; 8 :FEM<EK@FE @E FLI $ NFIB N?<E N< I8E @EKF JFD<K?@E> N< ;@;EcK LE;<IJK8E; N?@:? N8J F=K<E N< NFLC; J8P `D8>@:ag8J @E `3 D8>@:8CCP ;F<J 4agKF I<D@E; FLIeJ<CM<J K?8K here was an unsolved problem, a gap in our understanding.$K @J =8I 9<KK<I KF J8P `D8>@:a K?8E `:FDGC<O@KPa FI `<D<I><E:<aK?< C8KK<I NFI;J :I<8K< 8E @CCLJ@FE F= LE;<IJK8E;@E> 2@J<I KF J8P `D8>@:a 8E; C<8M< PFLIJ<C= 8 GC8:<?FC;<I 8 I<D@E;<I F= NFIB PFL

 N@CC?8M<KF;FC8K<I

+8><b/?<!LK@C@KPF=D<I><E:<c +8><b4FL8E!8:<-<8C@KPc +8><bFII<JGFE;<E:<@8Jc

+8><b"L<JJ@E>K?</<8:?<IcJ+8JJNFI;c +8><b"L<JJ@E>K?</<8:?<IcJ+8JJNFI;c

+1&',4'%($0!,15

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 164/309

21. Positive Bias: Look Into the Dark 

$ 8D K<8:?@E> 8 :C8JJ 8E; $ NI@K< LGFE K?< 9C8:B9F8I; K?I<< ELDe

9<IJ ee `$ 8D K?@EB@E> F= 8 ILC<a $ J8P `N?@:? >FM<IEJ J<eHL<E:<J F= K?I<< ELD9<IJ /?< J<HL<E:< ee 8J @K JF ?8GG<EJF9<PJ K?@J ILC< 8:? F= PFL N@CC =@E; FE PFLI ;<JB 8 G@C< F= @E;<O :8I;J 2I@K< ;FNE 8 J<HL<E:< F= K?I<< ELD9<IJ FE 8 :8I; 8E; $cCCD8IB @K `4<Ja =FI =@KJ K?< ILC< FI `)Fa =FI EFK =@KK@E> K?< ILC< /?<E

 PFL :8E NI@K< ;FNE 8EFK?<I J<K F= K?I<< ELD9<IJ 8E; 8JB N?<K?<I@K =@KJ 8>8@E 8E; JF FE 2?<E PFLcI< :FE=@;<EK K?8K PFL BEFN K?<ILC< NI@K< ;FNE K?< ILC< FE 8 :8I; 4FL :8E K<JK 8J D8EP KI@GC<KJ 8J

 PFLC@B<a

#<I<cJK?<I<:FI;F=FE<JKL;<EKcJ>L<JJ<J

)F 4<J 4<J

 K K?@J GF@EK K?< JKL;<EK NIFK< ;FNE ?@J >L<JJ 8K K?< ILC<2?8K ;F you K?@EB K?< ILC< @J 2FLC; PFL ?8M< N8EK<; KF K<JK 8EeFK?<I KI@GC<K 8E; @= JF N?8K NFLC; @K 9< /8B< 8 DFD<EK KF K?@EB9<=FI<:FEK@EL@E>

/?< :?8CC<E>< 89FM< @J 98J<; FE 8 :C8JJ@: <OG<I@D<EK ;L< KF+<K<I 28JFE K?< ee K8JB CK?FL>? JL9A<:KJ >@M<E K?@J K8JB KPGe@:8CCP <OGI<JJ<; ?@>? :FE=@;<E:< @E K?<@I >L<JJ<J FECP n F= K?<JL9A<:KJ JL::<JJ=LCCP >L<JJ<; K?< <OG<I@D<EK<IcJ I<8C ILC< 8E; I<GC@e:8K@FEJ J@E:< K?<E ?8M< :FEK@EL<; KF J?FN JL::<JJ I8K<J F= 8IFLE;n

/?< JKL;P N8J :8CC<; `*E K?< =8@CLI< KF <C@D@E8K< ?PGFK?<J<J @E8 :FE:<GKL8C K8JBa h Quarterly Journal of Experimental Psychology, e i .L9A<:KJ N?F 8KK<DGK K?< ee K8JB LJL8CCP KIP KF

 ><E<I8K< positive <O8DGC<J I8K?<I K?8E negative <O8DGC<JgK?<P 8GeGCP K?< ?PGFK?<K@:8C ILC< KF ><E<I8K< 8 I<GI<J<EK8K@M< @EJK8E:< 8E;J<<@=@K@JC89<C<;`4<Ja

/?LJ JFD<FE< N?F =FIDJ K?< ?PGFK?<J@J `ELD9<IJ @E:I<8J@E> 9P KNFa N@CC K<JK K?< KI@GC<K ee ?<8I K?8K @K =@KJ 8E; :FE=@;<EKCP 8EEFLE:< K?< ILC< .FD<FE< N?F =FIDJ K?< ?PGFK?<J@J 3e3e3

 N@CC K<JK K?< KI@GC<K ee ;@J:FM<I K?8K @K =@KJ 8E; K?<E 8EEFLE:<K?8KILC<

http://lesswrong.com/lw/iw/positive_bias_look_into_the_dark/ 

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 165/309

$E <M<IP :8J< K?< 8:KL8C ILC< @J K?< J8D< K?< K?I<< ELD9<IJ DLJK9<@E8J:<E;@E>FI;<I

LK KF ;@J:FM<I K?@J PFL NFLC; ?8M< KF ><E<I8K< KI@GC<KJ K?8K

 shouldn’t  =@K JL:? 8J ee 8E; J<< @= K?<P 8I< C89<C<; `)Fa2?@:? G<FGC< K<E; EFK KF ;F @E K?@J <OG<I@D<EK $E JFD< :8J<JJL9A<:KJ ;<M@J< `K<JKa 8E; 8EEFLE:< ILC<J =8I DFI< :FDGC@:8K<;K?8EK?<8:KL8C8EJN<I

/?@J :F>E@K@M< G?<EFD<EFE @J LJL8CCP CLDG<; @E N@K? `:FE=@IeD8K@FE 9@8Ja #FN<M<I @K J<<DJ KF D< K?8K K?< G?<EFD<EFE F= KIP@E> KF K<JK positive I8K?<I K?8E negative <O8DGC<J FL>?K KF 9< ;@JeK@E>L@J?<; =IFD K?< G?<EFD<EFE F= KIP@E> KF GI<J<IM< K?< 9<C@<= 

 PFL JK8IK<; N@K? `+FJ@K@M< 9@8Ja @J JFD<K@D<J LJ<; 8J 8 JPEFEPD

=FI`:FE=@ID8K@FE9@8Ja8E;=@KJK?@JG8IK@:LC8I=C8NDL:?9<KK<I

$K FE:< J<<D<; K?8K G?CF>@JKFE K?<FIP  :FLC; <OGC8@E 8 =C8D< >F@E> FLK @E 8E <E:CFJ<; 9FO hK?< 8@I 9<:8D< J8KLI8K<; N@K? G?CFe >@JKFE 8E; EF DFI< :FLC; 9< I<C<8J<;i 9LK G?CF>@JKFE K?<FIP :FLC;ALJK 8J N<CC ?8M< <OGC8@E<; K?< =C8D< not >F@E> FLK /F EFK@:< K?@J

 PFL ?8M< KF J<8I:? =FI E<>8K@M< <O8DGC<J @EJK<8; F= GFJ@K@M< <O8DeGC<J CFFB @EKF Q<IF @EJK<8; F= FE< N?@:? >F<J 8>8@EJK K?< >I8@E F= 

 N?8K<OG<I@D<EK?8JJ?FNEKF9<?LD8E@EJK@E:K

!FI9P@EJK@E:KN<?LD8E9<@E>JFECPC@M<@E?8C=K?<NFIC;*E< D8P 9< C<:KLI<; FE GFJ@K@M< 9@8J =FI ;8PJ 8E; P<K FM<ICFFB

@K @EeK?<eDFD<EK +FJ@K@M< 9@8J @J EFK JFD<K?@E> N< ;F 8J 8 D8KK<IF= CF>@: FI <M<E 8J 8 D8KK<I F= <DFK@FE8C 8KK8:?D<EK /?< eeK8JB @J `:FC;a CF>@:8C EFK 8==<:K@M<CP `?FKa E; P<K K?< D@JK8B<@J JL9eM<I98C FE K?< C<M<C F= @D8><IP F= @EJK@E:K@M< I<8:K@FEJ <e:8LJ< K?< GIF9C<D ;F<JEcK 8I@J< =IFD =FCCFN@E> 8 ;<C@9<I8K< ILC< K?8KJ8PJ `*ECP K?@EB 89FLK GFJ@K@M< <O8DGC<Ja @K :8EcK 9< JFCM<; ALJK 9P BEFN@E> M<I98CCP K?8K `2< FL>?K KF K?@EB 89FLK 9FK? GFJ@K@M< 8E;

E<>8K@M< <O8DGC<Ja 2?@:? <O8DGC< 8LKFD8K@:8CCP GFGJ @EKF PFLI?<8; 4FL ?8M< KF C<8IE NFI;C<JJCP KF Q8> @EJK<8; F= Q@> 4FL?8M<KFC<8IEKF=C@E:?KFN8I;K?<Q<IF@EJK<8;F=8N8P=IFD@K

$ ?8M< 9<<E NI@K@E> =FI HL@K< JFD< K@D< EFN FE K?< EFK@FE K?8KK?< JKI<E>K? F= 8 ?PGFK?<J@J @J N?8K @K can’t  <OGC8@E EFK N?8K @Kcan g@= PFL 8I< <HL8CCP >FF; 8K <OGC8@E@E> 8EP FLK:FD< PFL ?8M<Q<IF BEFNC<;>< .F KF JGFK 8E <OGC8E8K@FE K?8K @JEcK ?<CG=LC @KcJ EFK

+8><b!8B<8LJ8C@KPc +8><b4FLI.KI<E>K?8J8-8K@FE8C@JKc

('+!,!.!+$''# ! &,',*# 

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 166/309

<EFL>? KF K?@EB F= N?8K @K ;F<J <OGC8@E M<IP N<CCgPFL 8CJF ?8M< KFJ<8I:? =FI I<JLCKJ @K couldn’t <OGC8@E 8E; K?@J @J K?< KIL< JKI<E>K? F= K?<K?<FIP

.F $ J8@; 8CC K?@J 8E; K?<E P<JK<I;8P $ :?8CC<E><; K?< LJ<=LCE<JJF= `<D<I><E:<a 8J 8 :FE:<GK *E< :FDD<EK<I :@K<; JLG<I:FE;L:eK@M@KP 8E; =<IIFD8>E<K@JD 8J <O8DGC<J F= <D<I><E:< $ I<GC@<; K?8KEFEeJLG<I:FE;L:K@M@KP 8E; EFEe=<IIFD8>E<K@JD N<I< 8CJF <O8DGC<JF= <D<I><E:< N?@:? N8J K?< GIF9C<D LK 9< @K =8I =IFD D< KF:I@K@:@Q< K?< :FDD<EK<I <JG@K< ?8M@E> I<8; <OK<EJ@M<CP FE `:FEe=@ID8K@FE 9@8Ja $ ;@;EcK JGFK K?< `>FK:?8a @E K?< ee K8JB K?< =@IJKK@D< $ I<8; 89FLK @K $KcJ 8 JL9M<I98C 9C@EBeI<8:K@FE K?8K ?8J KF 9<I<KI8@E<; $cDJK@CCNFIB@E>FE@KDPJ<C=

.F DL:? F= 8 I8K@FE8C@JKcJ JB@CC @J 9<CFN K?< C<M<C F= NFI;J $KD8B<J =FI :?8CC<E>@E> NFIB @E KIP@E> KF :FEM<P K?< IK K?IFL>? 9CF> GFJKJ +<FGC< N@CC 8>I<< N@K? PFL 9LK K?<E @E K?< E<OK J<EK<E:<;F JFD<K?@E> JL9;<C@9<I8K@M< K?8K >F<J @E K?< FGGFJ@K< ;@I<:K@FE)FK K?8K $cD :FDGC8@E@E> D8AFI I<8JFE $cD GFJK@E> ?<I< @J KFF9J<IM<N?8KDPNFI;J haven’t :FEM<P<;

 I< PFL J<8I:?@E> =FI GFJ@K@M< <O8DGC<J F= GFJ@K@M< 9@8J I@>?KEFN FI JG8I@E> 8 =I8:K@FE F= PFLI J<8I:? FE N?8K GFJ@K@M< 9@8J

J?FLC;C<8;PFLKF not J<< @;PFLCFFBKFN8I;C@>?KFI;8IBE<JJ

+8><b/?<!LK@C@KPF=D<I><E:<c

%1+,*!'-+ &+/*+,'%1+,*!'-+)-+,!'&+

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 167/309

22. My Wild and Reckless Youth 

$K @J J8@; K?8K G8I<EKJ ;F 8CC K?< K?@E>J K?<P K<CC K?<@I :?@C;I<E EFK KF

;FN?@:?@J?FNK?<PBEFNEFKKF;FK?<D'FE> 8>F @E K?< LEK?@EB89CP ;@JK8EK G8JK $ N8J 8 ;<MFK<; /I8e

;@K@FE8C -8K@FE8C@JK :FE:<@M@E> DPJ<C= JB@CC<; 8::FI;@E> KF K?8KB@E; P<K $ BE<N EFK K?< 28P F= 8P<J 2?<E K?< PFLE> C@<Q<I

 N8J :FE=IFEK<; N@K? 8 DPJK<I@FLJeJ<<D@E> HL<JK@FE K?< GI<:<GKJF= /I8;@K@FE8C -8K@FE8C@KP ;@; EFK JKFG ?@D =IFD ;<M@J@E> 8 (PJK<eI@FLJ EJN<I $K @J 9P =8I K?< DFJK <D98II8JJ@E> D@JK8B< $ D8;<@EDPC@=<8E;$JK@CCN@E:<KFK?@EBF=@K

2?8K N8J DP DPJK<I@FLJ 8EJN<I KF 8 DPJK<I@FLJ HL<JK@FE /?@J$ N@CC EFK ;<J:I@9< =FI @K NFLC; 9< 8 CFE> K8C< 8E; :FDGC@:8K<; $ N8J PFLE> 8E; 8 D<I< /I8;@K@FE8C -8K@FE8C@JK N?F BE<N EFK K?<K<8:?@E>J F= /M<IJBP 8E; &8?E<D8E $ BE<N 89FLK *::8DcJ -8eQFI 9LK EFK K?< :FEALE:K@FE =8CC8:P  $ K?FL>?K $ :FLC; ><K 8N8P 

 N@K? K?@EB@E> :FDGC@:8K<; K?FL>?KJ DPJ<C= @E K?< C@K<I8IP JKPC< F= K?< :FDGC@:8K<; K?FL>?KJ $ I<8; @E J:@<E:< 9FFBJ EFK I<8C@Q@E> K?8K:FII<:K :FDGC<O@KP @J FECP GFJJ@9C< N?<E <M<IP JK<G @J G@EE<; ;FNEFM<IN?<CD@E>CP /F;8P FE< F= K?< :?@<= G@<:<J F= 8;M@:< $ >@M< KF

8JG@I@E> PFLE> I8K@FE8C@JKJ @J `F EFK 8KK<DGK CFE> :?8@EJ F= I<8eJFE@E>FI:FDGC@:8K<;GC8EJa

)FK?@E> DFI< K?8E K?@J E<<; 9< J8@; M<E 8=K<I $ @EM<EK<; DP `8EJN<Ia K?< G?<EFD<EFE N8J JK@CC 8 DPJK<IP  LEKF D< 8E; GFJeJ<JJ<; K?< J8D< HL8C@KP F= NFE;IFLJ @DG<E<KI89@C@KP K?8K @K ?8; 8KK?<JK8IK

(8B< EF D@JK8B< K?8K PFLE><I C@<Q<I N8J EFK JKLG@; CC K?<<IIFIJ F= N?@:? K?< PFLE> C@<Q<I N8J >L@CKP 8I< JK@CC 9<@E> D8;<

KF;8P 9P I<JG<:K<; J:@<EK@JKJ @E I<JG<:K<; AFLIE8CJ $K NFLC; ?8M<K8B<E 8 JL9KC<I JB@CC KF GIFK<:K ?@D K?8E <M<I ?< N8J K8L>?K 8J 8 /I8;@K@FE8C-8K@FE8C@JK

$E;<<; K?< PFLE> C@<Q<I ;@C@><EKCP 8E; G8@EJK8B@E>CP =FCCFN<;K?< @EALE:K@FEJ F= /I8;@K@FE8C -8K@FE8C@KP @E K?< :FLIJ< F= >F@E> 8JKI8P

http://lesswrong.com/lw/iy/my_wild_and_reckless_youth/  +8><b(PJK<I@FLJEJN<IJKF(PJK<I@FLJ,L<JK@FEJc http://en.wikipedia.org/wiki/Conjunction_fallacy

+8><b(PJK<I@FLJEJN<IJKF(PJK<I@FLJ,L<JK@FEJc +8><bFII<JGFE;<E:<@8Jc

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 168/309

 J 8 /I8;@K@FE8C -8K@FE8C@JK K?< PFLE> C@<Q<I N8J :8I<=LC KF<EJLI< K?8K ?@J (PJK<I@FLJ EJN<I D8;< 8 9FC; GI<;@:K@FE F= =LKLI<<OG<I@<E:< )8D<CP $ <OG<:K<; =LKLI< E<LIFCF>@JKJ KF ;@J:FM<I

K?8K E<LIFEJ N<I< <OGCF@K@E> HL8EKLD >I8M@KP 8 C8 .@I -F><I +<EeIFJ< /?@J I<HL@I<; E<LIFEJ KF D8@EK8@E 8 :<IK8@E ;<>I<< F= HL8EeKLD :F?<I<E:< N?@:? N8J JFD<K?@E> PFL :FLC; CFFB =FI 8E; =@E;FIEFK=@E; @K?<IPFLF9J<IM<K?8KFIPFL;FEcKI@>?K

LK DP ?PGFK?<J@J D8;< EF retrospective GI<;@:K@FEJ ::FI;@E> KF /I8;@K@FE8C .:@<E:< I<KIFJG<:K@M< GI<;@:K@FEJ ;FEcK :FLEKgJF

 N?P 9FK?<I D8B@E> K?<D /F 8 8P<J@8E FE K?< FK?<I ?8E; @= 8 ?PGFK?<J@J ;F<J EFK today ?8M< 8 =8MFI89C< C@B<C@?FF; I8K@F FM<I `$;FEcK BEFNa @K I8@J<J K?< HL<JK@FE F= N?P PFL today 9<C@<M< 8EPK?@E> 

DFI< :FDGC@:8K<; K?8E `$ ;FEcK BEFNa LK $ BE<N EFK K?< 28P F= 8P<J JF $ N8J EFK K?@EB@E> 89FLK C@B<C@?FF; I8K@FJ FI =F:LJ@E> GIF989@C@KP ;<EJ@KP $ ?8; (8;< 8 !8CJ@=@89C< +I<;@:K@FE N8J K?@JEFKK?<'8N

 J 8 /I8;@K@FE8C -8K@FE8C@JK K?< PFLE> C@<Q<I N8J :8I<=LC EFKKF 9<C@<M< @E D8>@: DPJK@:@JD :8I9FE :?8LM@E@JD FI 8EPK?@E> F= K?8K JFIK $ GIFL;CP GIF=<JJ<; F= DP (PJK<I@FLJ EJN<I `$K @J ALJKG?PJ@:J C@B< 8CC K?< I<JK F= G?PJ@:Ja J @= PFL :FLC; J8M< D8>@: =IFD

9<@E> 8 :F>E@K@M< @JFDFIG? F= D8>@: 9P :8CC@E> 

@K HL8EKLD >I8Me@KP LK $ BE<N EFK K?< 28P F= 8P<J 8E; ;@; EFK J<< K?< C<M<C

FE N?@:? DP @;<8 N8J @JFDFIG?@: KF D8>@: $ >8M< DP  allegianceKF G?PJ@:J 9LK K?@J ;@; EFK J8M< D< N?8K ;F<J GIF989@C@KP K?<FIP BEFN F= 8CC<>@8E:<J $ 8MF@;<; <M<IPK?@E> K?8K /I8;@K@FE8C -8K@FeE8C@KPKFC;D<N8J=FI9@;;<E9LKN?8KN8JC<=KN8JJK@CCD8>@:

<PFE; 8 ;FL9K DP 8CC<>@8E:< KF /I8;@K@FE8C -8K@FE8C@KP ?<CG<;D< ><K FLK F= K?< ?FC< $ ;L> DPJ<C= @EKF $= $ ?8;EcK 9<<E 8 /I8e;@K@FE8C -8K@FE8C@JK $ NFLC; ?8M< 9<<E completely J:I<N<; LK

/I8;@K@FE8C -8K@FE8C@KP JK@CC N8JEcK <EFL>? KF ><K @K right. $K ALJK C<;D< @EKF ;@==<I<EK D@JK8B<J K?8E K?< FE<J @K ?8; <OGC@:@KCP =FI9@;;<E

2?<E $ K?@EB 89FLK ?FN DP PFLE><I J<C= M<IP :8I<=LCCP =FCCFN<;K?< ILC<J F= /I8;@K@FE8C -8K@FE8C@KP @E K?< :FLIJ< F= ><KK@E> K?<8EJN<I wrong, @K J?<;J C@>?K FE K?< HL<JK@FE F= N?P G<FGC< N?F:8CC K?<DJ<CM<J `I8K@FE8C@JKJa ;F EFK ILC< K?< NFIC; 4FL E<<; one

+8><b+IF=<JJ@E>8E;?<<I@E>c +8><b.:@<E:<8JKK@I<c

+8><b!8B<OGC8E8K@FEJc +8><b&EFN@E>9FLK@8J<J8E#LIK+<FGC<c

%1+,*!'-+ &+/*+,'%1+,*!'-+)-+,!'&+

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 169/309

whole hell of a lot F= I8K@FE8C@KP 9<=FI< @K ;F<J 8EPK?@E> 9LK C<8; PFL@EKFE<N8E;@EK<I<JK@E>D@JK8B<J

/I8;@K@FE8C -8K@FE8C@KP @J K8L>?K 8J 8E 8IK I8K?<I K?8E 8 J:@<E:<

 PFL I<8; K?< 9@F>I8G?P F= =8DFLJ G?PJ@:@JKJ ;<J:I@9@E> K?< C<JJFEJC@=< K8L>?K K?<D 8E; PFL KIP KF ;F N?8K K?<P K<CC PFL KF ;F LK PFL?8M<EcK C@M<; K?<@I C@M<J 8E; ?8C= F= N?8K K?<PcI< KIP@E> KF ;<J:I@9<@J8E@EJK@E:KK?8K?8J9<<EKI8@E<;@EKFK?<D

/?< N8P /I8;@K@FE8C -8K@FE8C@KP @J ;<J@>E<; @K NFLC; ?8M< 9<<E8::<GK89C< =FI D< KF JG<E; P<8IJ FE DP J@CCP @;<8 JF CFE> 8J $ JL:e:<<;<; @E =8CJ@=P@E> @K <M<EKL8CCP 8E; N8J ?FE<JK N@K? DPJ<C= 89FLK

 N?8K DP K?<FIP GI<;@:K<; 8E; 8::<GK<; K?< ;@JGIFF= N?<E @K 8IeI@M<; <K :<K<I8 /?@J @J <EFL>? KF C<K K?< -8K:?<K F= .:@<E:< :C@:B

=FIN8I; 9LK @KcJ 8 C@KKC< ?8IJ? FE K?< G<FGC< N?F N8JK< P<8IJ F= K?<@I C@M<J /I8;@K@FE8C -8K@FE8C@KP @J 8 N8CB EFK 8 ;8E:< $KcJ ;<eJ@>E<; KF ><K PFL KF K?< KILK? eventually 8E; >@M<J PFL 8CC KFF DL:?K@D<KFJD<CCK?<=CFN<IJ8CFE>K?<N8P

/I8;@K@FE8C -8K@FE8C@JKJ :8E 8>I<< KF ;@J8>I<< /I8;@K@FE8C -8eK@FE8C@KP ;F<JEcK ?8M< K?< ideal K?8K K?@EB@E> @J 8E <O8:K 8IK @E N?@:?K?<I< @J FECP FE< :FII<:K GIF989@C@KP <JK@D8K< >@M<E K?< <M@;<E:<$E /I8;@K@FE8C -8K@FE8C@KP PFLcI< 8CCFN<; KF >L<JJ 8E; K?<E K<JK

 PFLI >L<JJ LK <OG<I@<E:< ?8J K8L>?K D< K?8K @= PFL ;FEcK know,8E;PFL>L<JJPFLcCC<E;LG9<@E>NIFE>

/?< 28P F= 8P<J @J 8CJF 8E @DGI<:@J< 8IK 8K C<8JK K?< N8P $cD ?FC;@E> =FIK? LGFE @K /?<J< 9CF> GFJKJ 8I< JK@CC =LD9C@E> 8KeK<DGKJ KF GLK @EKF NFI;J C<JJFEJ K?8K NFLC; 9< 9<KK<I K8L>?K 9P <OG<I@<E:< LK 8K C<8JK K?<I<cJ underlying  D8K? GCLJ <OG<I@D<EeK8C <M@;<E:< =IFD :F>E@K@M< GJP:?FCF>P FE ?FN ?LD8EJ 8:KL8CCP K?@EB (8P9< K?8K N@CC 9< <EFL>? KF :IFJJ K?< JKI8KFJG?<I@:8CCP ?@>? K?I<J?FC; I<HL@I<; =FI 8 ;@J:@GC@E< K?8K C<KJ PFL 8:KL8CCP ><K @K

I@>?K @EJK<8; F= ALJK :FEJKI8@E@E> PFL @EKF @EK<I<JK@E> E<N D@JK8B<J

%1/!$&* #$++1'-,

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 170/309

23. Failing to Learn from History 

Continuation of: (P2@C;8E;-<:BC<JJ4FLK?

*E:< LGFE 8 K@D< @E DP N@C; 8E; I<:BC<JJ PFLK? N?<E $ BE<N EFK K?< 28P F= 8P<J $ >8M< 8 (PJK<I@FLJ EJN<I KF 8 DPJK<I@FLJeJ<<D@E> HL<JK@FE (8EP =8@CLI<J F::LII<; @E J<HL<E:< 9LK FE<D@JK8B< JK8E;J FLK 8J DFJK :I@K@:8C (P PFLE><I J<C= ;@; EFK I<8Ce@Q< K?8K solving a mystery should make it feel less confusing. $ N8J KIP@E> KF <OGC8@E 8 (PJK<I@FLJ +?<EFD<EFEgN?@:? KF D< D<8EK GIFe

 M@;@E> 8 :8LJ< =FI @K =@KK@E> @K @EKF 8E @EK<>I8K<; DF;<C F= I<8C@KP2?P J?FLC; K?@J D8B< K?< G?<EFD<EFE C<JJ (PJK<I@FLJ N?<E K?8K

@J @KJ E8KLI< $ N8J KIP@E> KF explain K?< (PJK<I@FLJ +?<EFD<EFEEFK I<E;<I @K h9P JFD< @DGFJJ@9C< 8C:?<DPi @EKF 8 DLE;8E< G?<eEFD<EFE 8 G?<EFD<EFE K?8K NFLC;EcK <M<E :8CC FLK =FI 8E LELJL8C<OGC8E8K@FE@EK?<=@IJKGC8:<

 J 8 /I8;@K@FE8C -8K@FE8C@JK $ BE<N K?< ?@JKFI@:8C K8C<J F= 8JeKIFCF><IJ 8E; 8JKIFEFDP F= 8C:?<D@JKJ 8E; :?<D@JKIP F= M@K8C@JKJ8E; 9@FCF>P LK K?< (PJK<I@FLJ +?<EFD<EFE N8J EFK C@B< K?@J$K N8J JFD<K?@E>  new, JFD<K?@E> JKI8E><I JFD<K?@E> DFI< ;@=e

=@:LCK JFD<K?@E> K?8K FI;@E8IP J:@<E:< ?8; =8@C<; KF <OGC8@E =FI:<EKLI@<Jg 

e 8J @= JK8IJ 8E; D8KK<I 8E; C@=< ?8; EFK 9<<E DPJK<I@<J =FI ?LEe;I<;J F= P<8IJ 8E; K?FLJ8E;J F= P<8IJ =IFD K?< ;8NE F= ?LD8EK?FL>?KI@>?KLGLEK@CJ:@<E:<=@E8CCPJFCM<;K?<Dg 

2< C<8IE 89FLK 8JKIFEFDP 8E; :?<D@JKIP 8E; 9@FCF>P @E J:?FFC8E; @K J<<DJ KF LJ K?8K K?<J< D8KK<IJ ?8M< always been K?< GIFG<II<8CD F= J:@<E:< K?8K K?<P ?8M< never been DPJK<I@FLJ 2?<E J:@e<E:< ;8I<J KF :?8CC<E>< 8 E<N "I<8K +LQQC< K?< :?@C;I<E F= K?8K

 ><E<I8K@FE 8I< JB<GK@:8C =FI K?<P ?8M< E<M<I J<<E J:@<E:< <OGC8@EJFD<K?@E> K?8K feels DPJK<I@FLJ KF K?<D .:@<E:< @J FECP >FF; =FI<OGC8@E@E> scientific JL9A<:KJC@B<JK8IJ8E;D8KK<I8E;C@=<

$ K?FL>?K K?< C<JJFE F= ?@JKFIP N8J K?8K 8JKIFCF><IJ 8E; 8Ce:?<D@JKJ 8E; M@K8C@JKJ ?8; 8E @EE8K< :?8I8:K<I =C8N  8 K<E;<E:P 

http://lesswrong.com/lw/iz/failing_to_learn_from_history/  +8><b(P2@C;8E;-<:BC<JJ4FLK?c +8><b(P2@C;8E;-<:BC<JJ4FLK?c

+8><b(PJK<I@FLJEJN<IJKF(PJK<I@FLJ,L<JK@FEJc +8><bFII<JGFE;<E:<@8Jc

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 171/309

KFN8I; DPJK<I@8E@JD N?@:? C<; K?<D KF :FD< LG N@K? DPJK<I@FLJ<OGC8E8K@FEJ =FI EFEeDPJK<I@FLJ JL9A<:KJ LK JLI<CP @= 8 G?<eEFD<EFE I<8CCP  was M<IP N<@I; 8 N<@I; <OGC8E8K@FE D@>?K 9< @E

FI;<I$K N8J FECP 8=K<IN8I; N?<E $ 9<>8E KF J<< K?< DLE;8E< JKIL:e

KLI< @EJ@;< K?< DPJK<IP K?8K $ I<8C@Q<; N?FJ< J?F<J $ N8J JK8E;@E> @E *ECP K?<E ;@; $ I<8C@Q< ?FN I<8JFE89C< M@K8C@JD ?8; J<<D<; at the time ?FN surprising 8E; embarrassing ?8; 9<<E K?< LE@M<IJ<cJ I<eGCPF=`'@=<@JDLE;8E<8E;;F<JEFKE<<;8N<@I;<OGC8E8K@FEa

2< I<8; ?@JKFIP 9LK N< ;FEcK live @K N< ;FEcK experience @K$= FECP $ ?8; personally GFJKLC8K<; 8JKIFCF>@:8C DPJK<I@<J 8E; K?<E;@J:FM<I<; )<NKFE@8E D<:?8E@:J GFJKLC8K<; 8C:?<D@:8C DPJK<I@<J

8E; K?<E ;@J:FM<I<; :?<D@JKIP GFJKLC8K<; M@K8C@JK@: DPJK<I@<J 8E;K?<E ;@J:FM<I<; 9@FCF>P $ NFLC; ?8M< K?FL>?K F= DP (PJK<I@FLJ

 EJN<I8E;J8@;KFDPJ<C= No way am I falling for that again.

!$!&,'$*&*'%!+,'*1

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 172/309

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 173/309

:?<D@JKJ ;<M@J@E> D8>@:8C GIFG<IK@<J F= D8KK<I FI M@K8C@JKJ GFJKLC8Ke@E>8EFG8HL<`<C8EM@K8CaKF<OGC8@E8CCF=9@FCF>P

2?<E $ =@E8CCP I<8C@Q<; N?FJ< J?F<J $ N8J JK8E;@E> @E K?<I<

 N8J 8 JL;;<E J?F:B F= LE<OG<:K<; :FEE<:K@FE N@K? K?< G8JK $I<8C@Q<; K?8K K?< @EM<EK@FE 8E; ;<JKIL:K@FE F= M@K8C@JDgN?@:? $?8; FECP I<8; 89FLK @E 9FFBJg?8; actually happened to real people,

 N?F <OG<I@<E:<; @K DL:? K?< J8D< N8P $ <OG<I@<E:<; K?< @EM<EK@FE8E; ;<JKIL:K@FE F= DP FNE DPJK<I@FLJ 8EJN<I E; $ 8CJF I<8C@Q<;K?8K @= $ ?8; 8:KL8CCP  experienced  K?< G8JKg@= $ ?8; C@M<; K?IFL>?G8JK J:@<EK@=@: I<MFCLK@FEJ DPJ<C= I8K?<I K?8E I<8;@E> 89FLK K?<D @E?@JKFIP 9FFBJg$ GIF989CP NFLC; not  ?8M< D8;< K?< J8D< D@JK8B<8>8@E $ NFLC; EFK ?8M< :FD< LG N@K? another  DPJK<I@FLJ 8EJN<I

K?<=@IJKK?FLJ8E;C<JJFEJNFLC;?8M<?8DD<I<;?FD<K?<DFI8C.F h$ K?FL>?Ki KF =<<C JL==@:@<EKCP K?< =FI:< F= ?@JKFIP $ J?FLC;

KIP KF 8GGIFO@D8K< K?< K?FL>?KJ F= 8E C@<Q<I N?F had C@M<; K?IFL>??@JKFIPg$ J?FLC; KIP KF K?@EB 8J @= <M<IPK?@E> $ I<8; 89FLK @E?@JKFIP 9FFBJ ?8; 8:KL8CCP ?8GG<E<; KF D< h2@K? 8GGIFGI@8K<I<N<@>?K@E> =FI K?< 8M8@C89@C@KP 9@8J F= ?@JKFIP 9FFBJg$ J?FLC;I<D<D9<I 9<@E> 8 K?FLJ8E; G<8J8EKJ =FI <M<IP ILC<Ii $ J?FLC; @DeD<IJ< DPJ<C= @E ?@JKFIP @D8>@E< living  K?IFL>? <I8J $ FECP J8N 8J

@EBFEG8G<I2?P J?FLC; $ I<D<D9<I K?< 2I@>?K IFK?<IJc =@IJK =C@>?K $ N8J EFK K?<I< LK 8J 8 I8K@FE8C@JK :FLC; $ ;8I< KF not  I<D<D9<I N?<E K?< <M<EK 8:KL8CCP ?8GG<E<; $J K?<I< JF DL:? ;@==<I<E:< 9<eKN<<E J<<@E> 8E <M<EK K?IFL>? PFLI <P<JgN?@:? @J 8:KL8CCP 8 :8LJ8C:?8@E @EMFCM@E> I<=C<:K<; G?FKFEJ EFK 8 ;@I<:K :FEE<:K@FEg8E;J<<@E> 8E <M<EK K?IFL>? 8 ?@JKFIP 9FFB +?FKFEJ 8E; ?@JKFIP 9FFBJ9FK?;<J:<E;9P:8LJ8C:?8@EJ=IFDK?<<M<EK@KJ<C=

$ ?8; KF FM<I:FD< K?< =8CJ< 8DE<J@8 F= 9<@E> 9FIE 8K 8 G8IK@:LC8I

K@D< $ ?8; KF I<:8CCgD8B< 8M8@C89C< g  all K?< D<DFI@<J EFK ALJKK?< D<DFI@<J N?@:? 9P D<I< :F@E:@;<E:< 9<CFE><; KF DPJ<C= 8E;DPFNE<I8

/?<8IK?9<:8D<FC;<IF=8JL;;<E

/F DP =FID<I D<DFIP K?< 0E@K<; .K8K<J ?8; 8CN8PJ <O@JKe<;gK?<I< N8J E<M<I 8 K@D< N?<E K?<I< N8J EF 0E@K<; .K8K<J $ ?8;EFK I<D<D9<I<; LEK@C K?8K K@D< ?FN K?< -FD8E DG@I< IFJ< 8E;

+8><b!8@C@E>KF'<8IE=IFD#@JKFIPc http://en.wikipedia.org/wiki/Availability_heuristic

%#!&!+,'*1.!$$

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 174/309

9IFL>?K G<8:< 8E; FI;<I 8E; C8JK<; K?IFL>? JF D8EP :<EKLI@<J LEeK@C $ =FI>FK K?8K K?@E>J ?8; <M<I 9<<E FK?<IN@J< 8E; P<K K?< DG@I<=<CC 8E; 98I98I@8EJ FM<II8E DP :@KP 8E; K?< C<8IE@E> K?8K $ ?8; GFJe

J<JJ<; N8J CFJK /?< DF;<IE NFIC; 9<:8D< DFI< =I8>@C< KF DP <P<J@KN8JEFKK?<=@IJKDF;<IENFIC;

.F D8EP D@JK8B<J D8;< FM<I 8E; FM<I 8E; over 8>8@E 9<:8LJ< $;@;EFKI<D<D9<ID8B@E>K?<D@E<M<IP<I8$E<M<IC@M<;_

 E; KF K?@EB G<FGC< JFD<K@D<J NFE;<I @= FM<I:FD@E> 9@8J @J@DGFIK8EK

FEcK PFL I<D<D9<I ?FN D8EP K@D<J PFLI 9@8J<J ?8M< B@CC<; PFL 4FL ;FEcK $cM< EFK@:<; K?8K JL;;<E 8DE<J@8 F=K<E =FCCFNJ8 =8K8C D@JK8B< LK K8B< @K =IFD D< @K ?8GG<E<; $ I<D<D9<I $

 N8JEcKK?<I<

.F K?< E<OK K@D< PFL ;FL9K K?< JKI8E><E<JJ F= K?< =LKLI< I<eD<D9<I ?FN PFL N<I< 9FIE @E 8 ?LEK<Ie>8K?<I<I KI@9< K<E K?FLJ8E;

 P<8IJ 8>F N?<E EF FE< BE<N F= .:@<E:< 8K 8CC -<D<D9<I ?FN PFL N<I< J?F:B<; KF K?< ;<GK?J F= PFLI 9<@E> N?<E .:@<E:< <OGC8@E<;K?< >I<8K 8E; K<II@9C< J8:I<; DPJK<I@<J K?8K PFL FE:< I<M<I<; JF?@>?CP -<D<D9<I ?FN PFL FE:< 9<C@<M<; K?8K PFL :FLC; =CP 9P <8K@E> K?< I@>?K DLJ?IFFDJ 8E; K?<E PFL 8::<GK<; N@K? ;@J8Ge

GF@EKD<EK K?8K PFL NFLC; E<M<I =CP 8E; K?<E PFL =C<N -<D<D9<I?FN PFL ?8; 8CN8PJ K?FL>?K K?8K JC8M<IP N8J I@>?K 8E; GIFG<I 8E;K?<E PFL :?8E><; PFLI D@E; FEcK @D8>@E< ?FN PFL could  ?8M<GI<;@:K<; K?< :?8E>< =FI K?8K @J 8DE<J@8 Remember K?8K @E =8:K

 PFL ;@; EFK >L<JJ -<D<D9<I ?FN :<EKLIP 8=K<I :<EKLIP K?< NFIC;:?8E><;@EN8PJPFL;@;EFK>L<JJ

(8P9<K?<EPFLN@CC9<C<JJJ?F:B<;9PN?8K?8GG<EJE<OK

+8><b#@E;J@>?K9@8Jc

%1+,*!'-+ &+/*+,'%1+,*!'-+)-+,!'&+

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 175/309

25. Explain/Worship/Ignore?

Followup to .<D8EK@: .KFGJ@>EJ (PJK<I@FLJ EJN<IJ KF (PJK<e

I@FLJ,L<JK@FEJ

 J FLI KI@9< N8E;<IJ K?IFL>? K?< >I8JJC8E;J J<8I:?@E> =FI =IL@KKI<<J 8E; GI<P @K ?8GG<EJ <M<IP EFN 8E; K?<E K?8K N8K<I GFLIJ;FNE=IFDK?<JBP

`2?P ;F<J N8K<I JFD<K@D<J =8CC =IFD K?< JBPa $ 8JB K?< 9<8I;e<;N@J<D8EF=FLIKI@9<

#< K?@EBJ =FI 8 DFD<EK K?@J HL<JK@FE ?8M@E> E<M<I F::LII<; KF?@D 9<=FI< 8E; K?<E J8PJ `!IFD K@D< KF K@D< K?< JBP JG@I@KJ 98Ke

KC<8E;N?<EK?<P;FK?<@I9CFF;;I@GJ=IFDK?<JBPa`2?<I<;FK?<JBPJG@I@KJ:FD<=IFDa$8JB

#@J MF@:< ;IFGJ KF 8 N?@JG<I `!IFD K?< 9<=FI< K@D< !IFD K?<CFE>CFE>8>Fa

2?<E @K I8@EJ 8E; PFL ;FEcK BEFN N?P PFL ?8M< J<M<I8C FGeK@FEJ !@IJK PFL :FLC; J@DGCP EFK 8JB N?PgEFK =FCCFN LG FE K?<HL<JK@FE FI E<M<I K?@EB F= K?< HL<JK@FE @E K?< =@IJK GC8:< /?@J @JK?< $>EFI< :FDD8E; N?@:? K?< 9<8I;<; N@J< D8E FI@>@E8CCP J<e

C<:K<; .<:FE; PFL :FLC; KIP KF ;<M@J< JFD< JFIK F= <OGC8E8K@FEK?< OGC8@E :FDD8E; 8J K?< 9<8I;<; D8E ;@; @E I<JGFEJ< KF PFLI=@IJK HL<JK@FE /?@I; PFL :FLC; <EAFP K?< J<EJ8K@FE F= DPJK<I@FLJeE<JJgK?< 2FIJ?@G:FDD8E;

)FN 8J PFL 8I< 9FLE; KF EFK@:< =IFD K?@J JKFIP <8:? K@D< PFLJ<C<:K OGC8@E K?< 9<JKe:8J< J:<E8I@F @J K?8K PFL ><K 8E <OGC8E8K@FEJL:? 8J `JBP JG@I@KJa LK K?<E K?@J <OGC8E8K@FE @KJ<C= @J JL9A<:K KFK?< J8D< ;@C<DD8gOGC8@E 2FIJ?@G FI $>EFI< 8:? K@D< PFL?@K OGC8@E J:@<E:< >I@E;J =FI 8 N?@C< I<KLIEJ 8E <OGC8E8K@FE 8E;

K?<E 8EFK?<I ;@8CF> 9FO GFGJ LG J >FF; I8K@FE8C@JKJ N< =<<C ;LKPe9FLE; KF B<<G ?@KK@E> OGC8@E 9LK @K J<<DJ C@B< 8 IF8; K?8K ?8J EF<E;

4FL ?@K OGC8@E =FI C@=< 8E; ><K :?<D@JKIP PFL ?@K OGC8@E=FI :?<D@JKIP 8E; ><K 8KFDJ PFL ?@K OGC8@E =FI 8KFDJ 8E; ><K<C<:KIFEJ 8E; EL:C<@ PFL ?@K OGC8@E =FI EL:C<@ 8E; ><K HL8EKLD

http://lesswrong.com/lw/j2/explainworshipignore/ 

+8><b.<D8EK@:.KFGJ@>EJc +8><b(PJK<I@FLJEJN<IJKF(PJK<I@FLJ,L<JK@FEJc

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 176/309

:?IFDF;PE8D@:J 8E; HL8IBJ PFL ?@K OGC8@E =FI ?FN K?< HL8IBJ >FKK?<I<8E;><K98:BK?<@>8E>_

2< :8E ?@K OGC8@E =FI K?< @> 8E> 8E; N8@K N?@C< J:@<E:<

 >I@E;J K?IFL>? @KJ GIF:<JJ 8E; D8P9< JFD<;8P @K N@CC I<KLIE 8 G<Ie=<:KCP >FF; <OGC8E8K@FE LK K?<E K?8K N@CC ALJK 9I@E> LG 8EFK?<I;@8CF> 9FO .F @= N< :FEK@EL< CFE> <EFL>? N< DLJK :FD< KF 8  special ;@8CF> 9FO 8 new FGK@FE 8E OGC8E8K@FE /?8K )<<;J )F OeGC8E8K@FE 8 GC8:< N?<I< K?< :?8@E <E;Jg8E; K?@J D8P9< @J K?<FECP<OGC8E8K@FENFIK?BEFN@E>

/?<I<g$ALJK?@K 2FIJ?@G

)<M<I =FI><K K?8K K?<I< 8I< D8EP DFI< N8PJ KF NFIJ?@G JFD<eK?@E>K?8EC@>?K@E>:8E;C<J8IFLE;8E8CK8I

$= $c; J8@; `#L? K?8K ;F<J J<<D G8I8;FO@:8C $ NFE;<I ?FN K?<8GG8I<EK G8I8;FO @J I<JFCM<;a K?<E $ NFLC; ?8M< ?@K OGC8@E N?@:?;F<JJFD<K@D<JK8B<8N?@C<KFGIF;L:<8E8EJN<I

 E; @= K?< N?FC< @JJL< J<<DJ KF PFL LE@DGFIK8EK FI @II<C<M8EKFI @= PFLc; I8K?<I GLK F== K?@EB@E> 89FLK @K LEK@C KFDFIIFN K?8E PFL?8M<?@K $>EFI<

.<C<:KPFLIFGK@FEN@J<CP

%1+,*!'-+ &+/*+,'%1+,*!'-+)-+,!'&+

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 177/309

26. “Science” as Curiosity-Stopper

Followup to .<D8EK@: .KFGJ@>EJ (PJK<I@FLJ EJN<IJ KF (PJK<e

I@FLJ,L<JK@FEJ .8P)FKbFDGC<O@KPc

$D8>@E< K?8K $ @E =LCC M@<N F= C@M< K<C<M@J@FE :8D<I8J I8@J<;DP ?8E;J 8E; :?8EK<; abracadabra 8E; :8LJ<; 8 9I@CC@8EK C@>?K KF9< 9FIE =C8I@E> @E <DGKP JG8:< 9<PFE; DP FLKJKI<K:?<; ?8E;J$D8>@E< K?8K $ :FDD@KK<; K?@J 8:K F= 9C8K8EK LED@JK8B<89C< JFI:<IP LE;<I K?< =LCC JLG<IM@J@FE F= %8D<J -8E;@ 8E; 8CC JB<GK@:8C 8ID@<J(FJK G<FGC< $ K?@EB NFLC; 9< fairly curious 8J KF N?8K N8J >F@E> FE

LK EFN JLGGFJ< @EJK<8; K?8K $ ;FEcK >F FE K<C<M@J@FE $ ;F EFK N@J? KF J?8I< K?< GFN<I EFI K?< KILK? 9<?@E; @K $ N8EK KF B<<G DP JFI:<IP J<:I<K E; P<K $ 8CJF N8EK KF :8JK DP JG<CCJ N?<E<M<I 8E;

 N?<I<M<I $ GC<8J< $ N8EK KF :8JK DP 9I@CC@8EK =C8I< F= C@>?K JF K?8K$ :8E I<8; 8 9FFB FE K?< KI8@EgN@K?FLK 8EPFE< 9<:FD@E> :LI@FLJ$JK?<I<8JG<CCK?8KJKFGJ:LI@FJ@KP

4<J @E;<<; 2?<E<M<I 8EPFE< 8JBJ `#FN ;@; PFL ;F K?8Ka $ALJKJ8P`.:@<E:<a

$KcJ EFK 8 I<8C <OGC8E8K@FE JF DL:? 8J 8 :LI@FJ@KPeJKFGG<I $K;F<JEcK K<CC PFL N?<K?<I K?< C@>?K N@CC 9I@>?K<E FI =8;< :?8E>< :FCFI@E ?L< FI J8KLI8K@FE 8E; @K :<IK8@ECP ;F<JEcK K<CC PFL ?FN KF D8B< 8 J@D@C8I C@>?K PFLIJ<C= 4FL ;FEcK 8:KL8CCP know 8EPK?@E> DFI< K?8E

 PFL BE<N 9<=FI< $ J8@; K?< D8>@: NFI; LK PFL KLIE 8N8P J8K@Je=@<;K?8KEFK?@E>LELJL8C@J>F@E>FE

<KK<IP<KK?<J8D<KI@:BNFIBJN@K?8JK8E;8I;C@>?KJN@K:?

!C@G8JN@K:?8E;8C@>?K9LC9KLIEJFE 2?P

$E J:?FFC FE< @J K8L>?K K?8K K?< G8JJNFI; KF K?< C@>?K 9LC9 @J

`C<:KI@:@KPa P EFN $ ?FG< PFLcI< N8IP F= D8IB@E> K?< C@>?K 9LC9`LE;<IJKFF;a FE JL:? 8 98J@J F<J J8P@E> `C<:KI@:@KPa C<K PFL;F :8C:LC8K@FEJ K?8K N@CC :FEKIFC PFLI 8EK@:@G8K@FE F= <OG<I@<E:<

http://lesswrong.com/lw/j3/science_as_curiositystopper/  +8><b.<D8EK@:.KFGJ@>EJc +8><b(PJK<I@FLJEJN<IJKF(PJK<I@FLJ,L<JK@FEJc +8><b.8P)FK`FDGC<O@KPac +8><b!8B<OGC8E8K@FEJc +8><b.<D8EK@:.KFGJ@>EJc

+8><b"L<JJ@E>K?</<8:?<IcJ+8JJNFI;c +8><b"L<JJ@E>K?</<8:?<IcJ+8JJNFI;c

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 178/309

/?<I< @J 8K K?< C<8JK 8 >I<8K ;<8C DFI< KF C<8IE h+?PJ@:@JKJ J?FLC;@>EFI< K?@J G8I8>I8G? 8E; JL9JK@KLK< 8 GIF9C<D @E <MFCLK@FE8IP K?<eFIP  N?<I< K?< JL9JK8E:< F= K?< K?<FIP @J 8>8@E @E :8C:LC8K@FEJ K?8K

=<NG<FGC<BEFN?FNKFG<I=FIDi$= PFL K?FL>?K K?< C@>?K 9LC9 N8J scientifically inexplicable, @K

 NFLC; J<@Q< K?< entirety F= PFLI 8KK<EK@FE 4FL NFLC; ;IFG N?8K<Me<I<CJ<PFLN<I<;F@E>8E;=F:LJFEK?8KC@>?K9LC9

LK N?8K ;F<J K?< G?I8J< `J:@<EK@=@:8CCP <OGC@:89C<a D<8E $KD<8EJ K?8K JFD<FE< else BEFNJ ?FN K?< C@>?K 9LC9 NFIBJ 2?<E

 PFL 8I< KFC; K?< C@>?K 9LC9 @J `J:@<EK@=@:8CCP <OGC@:89C<a PFL ;FEcKBEFN DFI< K?8E PFL BE<N <8IC@<I PFL ;FEcK BEFN N?<K?<I K?< C@>?K9LC9 N@CC 9I@>?K<E FI =8;< LK 9<:8LJ< JFD<FE< else BEFNJ @K ;<e

 M8CL<JK?<BEFNC<;><@EPFLI<P<J 4FL9<:FD<C<JJ:LI@FLJ

.@E:< K?@J @J 8E <:FE9CF> JFD<FE< FLK K?<I< @J 9FLE; KF J8P `$= K?< C@>?K 9LC9 N<I< LEBEFNE KF J:@<E:< PFL :FLC; >8@E =8D< 8E;=FIKLE< 9P @EM<JK@>8K@E> @Ka LK $cD EFK K8CB@E> 89FLK >I<<; $cDEFK K8CB@E> 89FLK :8I<<I 8D9@K@FE $cD K8CB@E> 89FLK K?< I8N <DFeK@FE F= :LI@FJ@KPgK?< =<<C@E> F= 9<@E> @EKI@>L<; 2?P J?FLC; your :LI@FJ@KP 9< ;@D@E@J?<; 9<:8LJ< JFD<FE< else, EFK PFL BEFNJ ?FN K?< C@>?K 9LC9 NFIBJ $J K?@J EFK JG@K< $KcJ EFK <EFL>? =FI you KF

BEFNFK?<IG<FGC<DLJK8CJF9<@>EFI8EKFIPFLNFEcK9<?8GGP/?<I< 8I< >FF;J K?8K BEFNC<;>< D8P J<IM< 9<J@;<J :LI@FJ@KP

JL:? 8J K?< JF:@8C LK@C@KP F= K<:?EFCF>P !FI K?<J< @EJKILD<EK8C >FF;J @K D8KK<IJ N?<K?<I JFD< FK?<I <EK@KP @E CF:8C JG8:< 8CI<8;P BEFNJ LK=FIDPFNE:LI@FJ@KPN?PJ?FLC;@KD8KK<I

<J@;<J :FEJ@;<I K?< :FEJ<HL<E:<J @= PFL G<ID@K `.FD<FE< <CJ<BEFNJ K?< 8EJN<Ia KF =LE:K@FE 8J 8 :LI@FJ@KPeJKFGG<I *E< ;8P PFL

 N8CB @EKF PFLI C@M@E> IFFD 8E; J<< 8 >@8EK >I<<E <C<G?8EK J<<D@E>eCP?FM<I@E>@ED@;8@IJLIIFLE;<;9P8E8LI8F=J@CM<IC@>?K

`2?8KK?<?<:BaPFLJ8P

 E; 8 MF@:< :FD<J =IFD 89FM< K?< <C<G?8EK J8P@E> `.*(e*4 '-4 &)*2. 2#4 /#$. '+#)/ $.#-a

`*?a PFL J8P `@E K?8K :8J< E<M<I D@E;a 8E; N8CB FE KF K?<B@K:?<E

http://lesswrong.com/lw/kr/an_alien_god/  http://godescalc.wordpress.com/2012/06/24/overlooked-elephant/ 

%1+,*!'-+ &+/*+,'%1+,*!'-+)-+,!'&+

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 179/309

$ ;FEcK BEFN K?< >I8E; LE@=@<; K?<FIP =FI K?@J LE@M<IJ<cJ C8NJ F= G?PJ@:J $ 8CJF ;FEcK BEFN DL:? 89FLK ?LD8E 8E8KFDP N@K? K?<<O:<GK@FE F= K?< 9I8@E $ :FLC;EcK GF@EK FLK FE DP 9F;P N?<I< DP 

B@;E<PJ 8I< 8E; $ :8EcK I<:8CC F==?8E; N?8K DP C@M<I ;F<J h$ 8D EFKGIFL; F= K?@J C8J N@K? 8CC K?< D8K? $ E<<; KF JKL;P $cD EFK C@B<CP KFC<8IE8E8KFDP8EPK@D<JFFEi

.?FLC; $ JF =8I 8J curiosity @J :FE:<IE<; 9< DFI< @EKI@>L<; 9P DP @>EFI8E:< F= K?< LCK@D8K< C8NJ F= G?PJ@:J K?8E K?< =8:K K?8K $;FEcKBEFNDL:?89FLKN?8K>F<JFE@EJ@;<DPFNE9F;P

$= $ I8@J<; DP ?8E;J 8E; :8JK 8 C@>?K JG<CC PFL NFLC; 9< @EeKI@>L<; .?FLC; PFL 9< 8EP  less @EKI@>L<; 9P K?< M<IP =8:K K?8K$ I8@J<; DP ?8E;J 2?<E PFL I8@J< PFLI 8ID 8E; N8M< 8 ?8E;

8IFLE; K?@J 8:K F= N@CC @J :FFI;@E8K<; 9P h8DFE> FK?<I 9I8@E 8I<8Ji PFLI :<I<9<CCLD $ 9<K PFL ;FEcK BEFN ?FN K?< :<I<9<CCLD NFIBJ I  BEFN 8 C@KKC<gK?FL>? FECP K?< >IFJJ ;<K8@CJ EFK <EFL>? KF G<Ie=FID :8C:LC8K@FEJ_ 9LK JF N?8K 2?8K ;F<J K?8K D8KK<I @=  you;FEcK BEFN 2?P J?FLC; K?<I< 9< 8 ;FL9C< JK8E;8I; F= :LI@FJ@KP =FIJFI:<IP8E;?8E;DFK@FEJ

'FFB 8K PFLIJ<C= @E K?< D@IIFI F PFL BEFN N?8K PFLcI< CFFBe@E> 8K F PFL BEFN N?8K CFFBJ FLK =IFD 9<?@E; PFLI <P<J F

 PFL BEFN N?8K PFL 8I< .FD< F= K?8K 8EJN<I .:@<E:< BEFNJ 8E;JFD< F= @K .:@<E:< ;F<J EFK LK N?P J?FLC; K?8K ;@JK@E:K@FE D8KeK<IKFPFLI:LI@FJ@KP@= you ;FEcKBEFN

F PFL BEFN ?FN PFLI BE<<J NFIB F PFL BEFN ?FN PFLIJ?F<J N<I< D8;< F PFL BEFN N?P PFLI :FDGLK<I DFE@KFI

 >CFNJ FPFLBEFNN?PN8K<I@JN<K

/?< NFIC; 8IFLE; PFL @J =LCC F= GLQQC<J +I@FI@K@Q< @= PFL DLJKLK ;F EFK :FDGC8@E K?8K :IL<C .:@<E:< ?8J <DGK@<; K?< NFIC; F= DPJK<IP 2@K? I<8JFE@E> JL:? 8J K?8K $ :FLC; ><K PFL KF FM<ICFFB

8E<C<G?8EK@EPFLIC@M@E>IFFD

4+!&5+-*!'+!,1+,'((* 

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 180/309

27. Applause Lights

Followup to .<D8EK@: .KFGJ@>EJ 2< FEcK -<8CCP 28EK 4FLI

+8IK@:@G8K@FE

 K K?< .@E>LC8I@KP .LDD@K FE< F= K?< JG<8B<IJ :8CC<; =FI;<DF:I8K@: DLCK@E8K@FE8C ;<M<CFGD<EK F= $ .F $ JK<GG<; LG KFK?<D@:IFG?FE<8E;8JB<;

.LGGFJ<K?8K8>IFLGF=;<DF:I8K@:I<GL9C@:J=FID8 :FEJFIK@LDKF;<M<CFG$8E;K?<I<cJ8CFKF=GFC@K@:B@E> ;LI@E>K?<GIF:<JJgJFD<@EK<I<JK>IFLGJ?8M<LELJL8CCP 

C8I><@E=CL<E:<FK?<IJ><KJ?8=K<;g@EFK?<INFI;JK?<I<JLCKCFFBJALJKC@B<K?<GIF;L:KJF=DF;<IE;<DF:I8:@<J CK<IE8K@M<CPJLGGFJ<8>IFLGF=I<9<CE<I;J;<M<CFGJ8E$@EK?<@I98J<D<EK8E;@EJKIL:KJK?<

 $KFGFCC<M<IPFE<@EK?<NFIC;g;IFGG@E>:<CCG?FE<JKF8EPFE<N?F;F<JEcK?8M<K?<Dg8E;;FN?8K<M<IK?<D8AFI@KPJ8PJ 2?@:?F=K?<J<;FPFLK?@EB@JDFI<`;<DF:I8K@:a8E;NFLC;PFL=<<CJ8=<N@K?<@K?<I

$ N8EK<; KF =@E; FLK N?<K?<I ?< 9<C@<M<; @E K?< GI8>D8K@: 8;e<HL8:P F= K?< ;<DF:I8K@: GFC@K@:8C GIF:<JJ FI @= ?< 9<C@<M<; @E K?<DFI8CI@>?KE<JJF=MFK@E> LKK?<JG<8B<II<GC@<;

/?<=@IJKJ:<E8I@FJFLE;JC@B<8E<;@KFI@8C@E-<8JFED8>8Q@E<8E;K?<J<:FE;JFLE;JC@B<8#FCCPNFF;DFM@<GCFK

FE=LJ<;$8JB<;

/?<EN?8KB@E;F=;<DF:I8K@:GIF:<JJ did PFL?8M<@ED@E;

/?<JG<8B<II<GC@<;

http://lesswrong.com/lw/jb/applause_lights/ 

+8><b.<D8EK@:.KFGJ@>EJc http://lesswrong.com/lw/ja/we_dont_really_want_your_participation/ 

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 181/309

.FD<K?@E>C@B<K?<#LD8E"<EFD<+IFA<:KgK?8KN8J8E@EK<IE8K@FE8CCPJGFEJFI<;I<J<8I:?GIFA<:K

$8JB<;

#FNNFLC;;@==<I<EK@EK<I<JK>IFLGJI<JFCM<K?<@I:FE=C@:KJ@E8JKIL:KLI<C@B<K?<#LD8E"<EFD<+IFA<:K

 E;K?<JG<8B<IJ8@;

$;FEcKBEFN

/?@J <O:?8E>< GLKJ D< @E D@E; F= 8 HLFK<

hN?@:? $ =8@C<; KF"FF>C< =FLE; 9P %<== "I<P 8E; (@>L<Ci =IFD JFD< ;@:K8KFI FI FK?e<I N?F N8J 8JB<; @= ?< ?8; 8EP @EK<EK@FEJ KF DFM< ?@J G<K JK8K<KFN8I;;<DF:I8:P

2<9<C@<M<N<8I<8CI<8;PN@K?@E8;<DF:I8K@:JPJK<D.FD<=8:KFIJ8I<JK@CCD@JJ@E>C@B<K?<<OGI<JJ@FEF=K?<G<FGC<cJN@CC

/?< JL9JK8E:< F= 8 ;<DF:I8:P @J K?< JG<:@=@: D<:?8E@JD K?8K I<eJFCM<J GFC@:P :FE=C@:KJ $= 8CC >IFLGJ ?8; K?< J8D< GI<=<II<; GFC@:@<JK?<I< NFLC; 9< EF E<<; =FI ;<DF:I8:PgN< NFLC; 8LKFD8K@:8CCP :FeFG<I8K< /?< I<JFCLK@FE GIF:<JJ :8E 9< 8 ;@I<:K D8AFI@KP MFK< FI 8E<C<:K<; C<>@JC8KLI< FI <M<E 8 MFK<IeJ<EJ@K@M< 9<?8M@FI F= 8E $ 9LK@K ?8J KF 9< something. 2?8K ;F<J @K mean KF :8CC =FI 8 `;<DF:I8K@:aJFCLK@FE @= PFL ;FEcK ?8M< 8 :FE=C@:KeI<JFCLK@FE D<:?8E@JD @E D@E;

$ K?@EB @K D<8EJ K?8K PFL ?8M< J8@; K?< NFI; `;<DF:I8:Pa JFK?< 8L;@<E:< @J JLGGFJ<; KF :?<<I $KcJ EFK JF DL:? 8  proposition-

 al JK8K<D<EK 8J K?< <HL@M8C<EK F= K?< `GGC8LJ<a C@>?K K?8K K<CCJ 8 JKL;@F8L;@<E:<N?<EKF:C8G

/?@J :8J< @J I<D8IB89C< FECP @E K?8K $ D@JKFFB K?< 8GGC8LJ<C@>?K =FI 8 GFC@:P JL>><JK@FE N@K? JL9J<HL<EK <D98II8JJD<EK =FI8CC (FJK 8GGC8LJ< C@>?KJ 8I< DL:? DFI< 9C8K8EK 8E; :8E 9< ;<K<:Ke<;9P8J@DGC<I<M<IJ8CK<JK !FI<O8DGC<JLGGFJ<JFD<FE<J8PJ

2<E<<;KF98C8E:<K?<I@JBJ8E;FGGFIKLE@K@<JF=$

http://www.time.com/time/magazine/article/0,9171,954853,00.html 

(($-+$!,+

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 182/309

$=PFLI<M<IJ<K?@JJK8K<D<EKPFL><K

2<J?FLC;EcK98C8E:<K?<I@JBJ8E;FGGFIKLE@K@<JF=$

.@E:< K?< I<M<IJ8C JFLE;J abEFID8C K?< LEI<M<IJ<; JK8K<D<EK@J GIF989CP EFID8C @DGCP@E> @K ;F<J EFK :FEM<P E<N @E=FID8K@FE/?<I< 8I< GC<EKP F= C<>@K@D8K< I<8JFEJ =FI LKK<I@E> 8 J<EK<E:< K?8K

 NFLC; 9< LE@E=FID8K@M< @E @JFC8K@FE `2< E<<; KF 98C8E:< K?< I@JBJ8E; FGGFIKLE@K@<J F= $a :8E @EKIF;L:< 8 ;@J:LJJ@FE KFG@: @K :8E<DG?8J@Q< K?< @DGFIK8E:< F= 8 JG<:@=@: GIFGFJ8C =FI 98C8E:@E> @K:8E :I@K@:@Q< 8E LE98C8E:<; GIFGFJ8C '@EB@E> KF 8 EFID8C 8JJ<IK@FE:8E :FEM<P E<N @E=FID8K@FE KF 8 9FLE;<; I8K@FE8C@JKgK?< C@EB @Ke

J<C= D8P EFK 9< F9M@FLJ LK @= no JG<:@=@:J =FCCFN K?< J<EK<E:< @JGIF989CP8E8GGC8LJ<C@>?K

$ 8D K<DGK<; KF >@M< 8 K8CB JFD<K@D< K?8K :FEJ@JKJ F= nothing but 8GGC8LJ< C@>?KJ 8E; J<< ?FN CFE> @K K8B<J =FI K?< 8L;@<E:< KF JK8IKC8L>?@E>

$8D?<I<KFGIFGFJ<KFPFLKF;8PK?8KN<E<<;KF98C8E:<K?<I@JBJ8E;FGGFIKLE@K@<JF=8;M8E:<;IK@=@:@8C$EK<CC@><E:< 2<J?FLC;8MF@;K?<I@JBJ8E;@EJF=8I8J@K

@JGFJJ@9C<I<8C@Q<K?<FGGFIKLE@K@<J 2<J?FLC;EFKE<<;C<JJCP:FE=IFEK<EK@I<CPLEE<:<JJ8IP;8E><IJ /F8:?@<M<K?<J<>F8CJN<DLJKGC8EN@J<CP8E;I8K@FE8CCP2<J?FLC;EFK8:K@E=<8I8E;G8E@:FI>@M<@EKFK<:?EFG?F9@89LKE<@K?<IJ?FLC;N<8:K@E9C@E;<EK?LJ@8JD 2<J?FLC;I<JG<:KK?<@EK<I<JKJF=8CCG8IK@<JN@K?8JK8B<@EK?<.@E>LC8I@KP 2<DLJKKIPKF<EJLI<K?8KK?<9<E<=@KJF=8;M8E:<;K<:?EFCF>@<J8::IL<KF8JD8EP@E;@M@;L8CJ8JGFJJ@9C<I8K?<IK?8E9<@E> I<JKI@:K<;KF8=<N 2<DLJKKIPKF8MF@;8JDL:?8JGFJJ@9C<M@FC<EK:FE=C@:KJLJ@E>K?<J<K<:?EFCF>@<J8E;

 N<DLJKGI<M<EKD8JJ@M<;<JKIL:K@M<:8G89@C@KP=IFD=8CC@E>@EKFK?<?8E;JF=@E;@M@;L8CJ 2<J?FLC;K?@EBK?IFL>?K?<J<@JJL<J9<=FI<EFK8=K<I@K@JKFFC8K<KF;F8EPK?@E>89FLKK?<D_

%1+,*!'-+ &+/*+,'%1+,*!'-+)-+,!'&+

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 183/309

28. Truly Part Of You

Followup to "L<JJ@E> K?< /<8:?<IcJ +8JJNFI; IK@=@:@8C ;;@e

K@FE

  :C8JJ@: G8G<I 9P I<N (:<IDFKK `IK@=@:@8C $EK<CC@><E:<(<<KJ )8KLI8C .KLG@;@KP ` :I@K@:@Q<; $ GIF>I8DJ K?8K NFLC; KIP KF I<GI<J<EK EFK@FEJ C@B< happiness is a state of mind LJ@E> 8 J<D8EK@:E<KNFIB

,--(!&$'

4

P$,P

#))$',,

 E; F= :FLIJ< K?<I<cJ EFK?@E>  inside K?< `#++$)..a EF;<@KcJALJK8E8B<;'$.+KFB<EN@K?8JL>><JK@M<E>C@J?E8D<

.F (:<IDFKK J8PJ ` >FF; K<JK =FI K?< ;@J:@GC@E<; GIF>I8DeD<I @J KF KIP LJ@E> ><EJPDJ @E B<P GC8:<J 8E; J<< @= ?< JK@CC 8;D@I<J?@J JPJK<D !FI <O8DGC< @= .//e*!e($) @J I<E8D<;

"_a K?<E N< NFLC; ?8M< $,#))$',, "`N?@:?CFFBJDL:?DFI<;L9@FLJa

*I 8J $ NFLC; JC@>?KCP I<G?I8J< K?< @;<8 $= PFL JL9JK@KLK<; I8Ee;FD@Q<; JPD9FCJ =FI all  K?< JL>><JK@M< E>C@J? E8D<J PFL NFLC;9< :FDGC<K<CP LE89C< KF =@>LI< FLK N?8K ""

D<8EK 28J K?< $ GIF>I8D D<8EK KF I<GI<J<EK ?8D9LI><IJ GeGC<J #8GG@E<JJ 2?F BEFNJ If you delete the suggestive English names, they don’t grow back.

.LGGFJ< 8 G?PJ@:@JK K<CCJ PFL K?8K `'@>?K @J N8M<J` 8E; PFL be- lieve ?@D 4FL EFN ?8M< 8 C@KKC< E<KNFIB @E PFLI ?<8; K?8K J8PJ$.eh'$"#/ 21.i $= JFD<FE< 8JBJ PFL `2?8K @J C@>?K D8;<F=aPFLcCC9<89C<KFJ8P`28M<Ja

 J (:<IDFKK J8PJ `/?< N?FC< GIF9C<D @J ><KK@E> K?< ?<8I<IKF EFK@:< N?8K @K ?8J 9<<E KFC; )FK bLE;<IJK8E;c 9LK bEFK@:<ca

http://lesswrong.com/lw/la/truly_part_of_you/  +8><b"L<JJ@E>K?</<8:?<IcJ+8JJNFI;c http://lesswrong.com/lw/l9/artificial_addition/ 

http://ontolog.cim3.net/forum/ontolog-forum/2006-04/pdfb9P8wB8aYh.pdf  +8><b"L<JJ@E>K?</<8:?<IcJ+8JJNFI;c

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 184/309

.LGGFJ< K?8K @EJK<8; K?< G?PJ@:@JK KFC; PFL `'@>?K @J D8;< F= C@KKC<:LIMP K?@E>Ja h)FK KIL< 9KNi 2FLC; PFL notice 8EP ;@==<I<E:< F= 8EK@:@G8K<;<OG<I@<E:<

#FN :8E PFL I<8C@Q< K?8K PFL J?FLC;EcK KILJK PFLI J<<D@E> BEFNC<;>< K?8K C@>?K @J N8M<Ja *E< K<JK PFL :FLC; 8GGCP @J 8JB@E>`FLC; $ regenerate K?@J BEFNC<;>< @= @K N<I< JFD<?FN ;<C<K<; =IFDDPD@E;a

/?@J @J J@D@C8I @E JG@I@K KF J:I8D9C@E> K?< E8D<J F= JL>><JK@M<CP E8D<; '$.+ KFB<EJ @E PFLI $ GIF>I8D 8E; J<<@E> @= JFD<FE< <CJ<:8E =@>LI< FLK N?8K K?<P 8CC<><;CP `I<=<Ia KF $KcJ 8CJF J@D@C8I @E JG@Ie@K KF F9J<IM@E> K?8K N?@C< 8E IK@=@:@8C I@K?D<K@:@8E :8E I<:FI;8E; GC8P 98:B )@IG(;,:J:B ,>L -=>FH::B @K :8EcK I<e

 ><E<I8K< K?< BEFNC<;>< @= PFL ;<C<K< @K =IFD D<DFIP LEK@C 8EFK?<I?LD8E I<e<EK<IJ @K @E K?< ;8K898J< %LJK 8J @= PFL =FI>FK K?8K `C@>?K@J N8M<Ja PFL :FLC;EcK ><K 98:B K?< BEFNC<;>< <O:<GK K?< J8D< N8P 

 PFL >FK K?< BEFNC<;>< KF 9<>@E N@K?g9P 8JB@E> 8 G?PJ@:@JK 4FL:FLC;EcK ><E<I8K< K?< BEFNC<;>< =FI PFLIJ<C= K?< N8P K?8K G?PJ@e:@JKJFI@>@E8CCP><E<I8K<;@K

/?< J8D< <OG<I@<E:<J K?8K C<8; LJ KF =FIDLC8K< 8 9<C@<= :FEE<:KK?8K 9<C@<= KF FK?<I BEFNC<;>< 8E; J<EJFIP @EGLK 8E; DFKFI FLKe

GLK $= PFL J<< 8 9<8M<I :?<N@E> 8 CF> K?<E PFL BEFN N?8K K?@JK?@E>eK?8Ke:?<NJeK?IFL>?eCF>J CFFBJ C@B< 8E; PFL N@CC 9< 89C< KFI<:F>E@Q< @K FE =LKLI< F::8J@FEJ N?<K?<I @K @J :8CC<; 8 `9<8M<Ia FIEFK LK @= PFL 8:HL@I< PFLI 9<C@<=J 89FLK 9<8M<IJ 9P JFD<FE< <CJ<K<CC@E> PFL =8:KJ 89FLK `9<8M<IJa PFL D8P EFK 9< 89C< KF I<:F>E@Q<89<8M<IN?<EPFLJ<<FE<

/?@J @J K?< K<II@9C< ;8E><I F= KIP@E> KF tell  8E IK@=@:@8C $EK<CC@e ><E:< =8:KJ N?@:? @K :FLC; EFK C<8IE =FI @KJ<C= $K @J 8CJF K?< K<II@9C<;8E><I F= KIP@E> KF tell JFD<FE< 89FLK G?PJ@:J K?8K K?<P :8EEFK M<Ie

@=P =FI K?<DJ<CM<J !FI N?8K G?PJ@:@JKJ D<8E 9P `N8M<a @J EFK `C@KKC<JHL@>>CPK?@E>a9LK8GLI<CPD8K?<D8K@:8C:FE:<GK

 J 8M@;JFE F9J<IM<J @= PFL 9<C@<M< K?8K `9<8M<IJa C@M< @E;<J<IKJ 8I< GLI< N?@K< @E :FCFI 8E; N<@>? GFLE;J N?<E 8;LCKK?<E PFL ;F EFK ?8M< 8EP 9<C@<=J about 9<8M<IJ KIL< FI =8CJ< 4FLI9<C@<= 89FLK `9<8M<IJa @J EFK I@>?K <EFL>? KF 9< NIFE> $= PFL ;FEcK?8M< <EFL>? <OG<I@<E:< KF I<><E<I8K< 9<C@<=J N?<E K?<P 8I< ;<C<Ke

+8><b(8B@E><C@<=J+8P-<EKh@EEK@:@G8K<;OG<I@<E:<Jic http://lesswrong.com/lw/l9/artificial_addition/ 

%1+,*!'-+ &+/*+,'%1+,*!'-+)-+,!'&+

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 185/309

<; K?<E ;F PFL ?8M< <EFL>? <OG<I@<E:< KF :FEE<:K K?8K 9<C@<= KF8EPK?@E> 8K 8CC 2@KK><EJK<@E ` N?<<C K?8K :8E 9< KLIE<; K?FL>?EFK?@E>KLIEJN@K?@K@JEFKG8IKF=K?<D<:?8E@JDa

 CDFJK 8J JFFE 8J $ JK8IK<; I<8;@E> 89FLK $g<M<E 9<=FI< $I<8; (:<IDFKKg$ I<8C@Q<; @K NFLC; 9< a really good idea KF 8CN8PJ8JB DPJ<C= `#FN NFLC; $ I<><E<I8K< K?@J BEFNC<;>< @= @K N<I<;<C<K<;=IFDDPD@E;a

/?< ;<<G<I K?< ;<C<K@FE K?< JKI@:K<I K?< K<JK $= 8CC GIFF=J F= K?< +PK?8>FI<8E /?<FI<D N<I< ;<C<K<; =IFD DP D@E; :FLC; $ I<eGIFM< @K $ K?@EB JF $= 8CC BEFNC<;>< F= K?< +PK?8>FI<8E /?<FI<D

 N<I< ;<C<K<; =IFD DP D@E; NFLC; $ EFK@:< K?< +PK?8>FI<8E /?<eFI<D KF I<eGIFM< /?8KcJ ?8I;<I KF 9F8JK N@K?FLK GLKK@E> @K KF K?<

K<JK 9LK @= PFL ?8E;<; D< 8 I@>?K KI@8E>C< N@K? J@;<J 8E; 8E;KFC; D< K?8K K?< C<E>K? F= K?< ?PGFK<ELJ< N8J :8C:LC89C< $ K?@EB $

 NFLC;9<89C<KF:8C:LC8K<@K@=$JK@CCBE<N8CCK?<I<JKF=DPD8K?

2?8K 89FLK K?< EFK@FE F= mathematical proof?  $= EF FE< ?8; <M<IKFC; @K KF D< NFLC; $ 9< 89C< KF I<@EM<EK that FE K?< 98J@J F= FK?e<I 9<C@<=J $ GFJJ<JJ /?<I< N8J 8 K@D< N?<E ?LD8E@KP ;@; EFK ?8M<JL:? 8 :FE:<GK .FD<FE< DLJK ?8M< @EM<EK<; @K 2?8K N8J @K K?8KK?<P EFK@:<; 2FLC; $ EFK@:< @= $ J8N JFD<K?@E> <HL8CCP EFM<C 8E;

<HL8CCP @DGFIK8EK 2FLC; $ 9< 89C< KF K?@EB K?8K =8I FLKJ@;< K?<9FO 

#FN DL:? F= PFLI BEFNC<;>< :FLC; PFL I<><E<I8K< !IFD ?FN ;<<G 8 ;<C<K@FE $KcJ EFK ALJK 8 K<JK KF :8JK FLK @EJL==@:@<EKCP :FEeE<:K<; 9<C@<=J $KcJ 8 N8P F= 89JFI9@E>  a fountain of knowledge, not just one fact.

  J?<G?<I; 9L@C;J 8 :FLEK@E> JPJK<D K?8K NFIBJ 9P K?IFN@E> 8 G<99C< @EKF 8 9L:B<K N?<E<M<I 8 J?<<G C<8M<J K?< =FC; 8E; K8B@E> 8 G<99C< FLK N?<E<M<I 8 J?<<G I<KLIEJ $= PFL K?< 8GGI<EK@:< ;FEFK LE;<IJK8E; K?@J JPJK<Dg@= @K @J D8>@: K?8K NFIBJ =FI EF 8GG8Ie<EK I<8JFEgK?<E PFL N@CC EFK BEFN N?8K KF ;F @= PFL 8::@;<EK8CCP ;IFG 8E <OKI8 G<99C< @EKF K?< 9L:B<K /?8K N?@:? PFL :8EEFK D8B<

 PFLIJ<C= PFL :8EEFK remake N?<E K?< J@KL8K@FE :8CCJ =FI @K 4FL :8EeEFK >F 98:B KF K?< JFLI:< KN<8B FE< F= K?< G8I8D<K<I J<KK@E>J 8E;I<><E<I8K< K?< FLKGLK N@K?FLK K?< JFLI:< $= `/NF GCLJ =FLI <HL8CJJ@Oa @J 8 9ILK< =8:K LEKF PFL 8E; K?<E FE< F= K?< <C<D<EKJ :?8E><J

+8><b/?<`*LKJ@;<K?<FOaFOc http://sl4.org/wiki/TheSimpleTruth

,*-$1(*,'1'-

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 186/309

KF `=@M<a ?FN 8I< PFL KF BEFN K?8K `KNF GCLJ =@M< <HL8CJ J<M<Ea N?<EPFLN<I<J@DGCP told K?8K`KNFGCLJ=FLI<HL8CJJ@Oa

$= PFL J<< 8 JD8CC GC8EK K?8K ;IFGJ 8 J<<; N?<E<M<I 8 9@I; G8JJ<J

@K @K N@CC EFK F::LI KF PFL K?8K PFL :8E LJ< K?@J GC8EK KF G8IK@8CCP 8LeKFD8K< K?< J?<<Ge:FLEK<I /?FL>? PFL C<8IE<; JFD<K?@E> K?8K K?<FI@>@E8C D8B<I NFLC; LJ< KF @DGIFM< FE ?@J @EM<EK@FE PFL :8EcK >F98:BKFK?<JFLI:<8E;I<e:I<8K<@K

2?<E PFL :FEK8@E K?< JFLI:< F= 8 K?FL>?K K?8K K?FL>?K :8E:?8E>< 8CFE> N@K? PFL 8J PFL 8:HL@I< E<N BEFNC<;>< 8E; E<N JB@CCJ 2?<E PFL :FEK8@E K?< JFLI:< F= 8 K?FL>?K @K 9<:FD<J KILCP 8G8IKF=PFL8E;>IFNJ8CFE>N@K?PFL

.KI@M< KF D8B< PFLIJ<C= K?< JFLI:< F= <M<IP K?FL>?K NFIK? K?@EBe@E> $= K?< K?FL>?K FI@>@E8CCP :8D< =IFD FLKJ@;< D8B< JLI< @K:FD<J =IFD @EJ@;< 8J N<CC FEK@EL8CCP 8JB PFLIJ<C= `#FN NFLC; $I<><E<I8K< K?< K?FL>?K @= @K N<I< ;<C<K<;a 2?<E PFL ?8M< 8E 8EeJN<I @D8>@E< that BEFNC<;>< 9<@E> ;<C<K<; 8J N<CC E; N?<E PFL=@E;8=FLEK8@EJ<<N?8K<CJ<@K:8EGFLI

%1+,*!'-+ &+/*+,'%1+,*!'-+)-+,!'&+

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 187/309

29. Chaotic Inversion 

$ N8J I<:<EKCP ?8M@E> 8 :FEM<IJ8K@FE N@K? JFD< =I@<E;J FE K?<

KFG@: F= ?FLIe9Pe?FLI GIF;L:K@M@KP 8E; N@CCGFN<I D8@EK<eE8E:<gJFD<K?@E>$cM<JKIL>>C<;N@K?DPN?FC<C@=<

$ :8E 8MF@; ILEE@E> 8N8P =IFD 8 ?8I; GIF9C<D K?< =@IJK K@D< $J<< @K hG<IJ<M<I8E:< FE 8 K@D<J:8C< F= J<:FE;Ji 8E; $ :8E JK@:B KFK?< J8D< GIF9C<D =FI P<8IJ 9LK KF B<<G NFIB@E> FE 8 K@D<J:8C< F=  hours @J 8 :FEJK8EK 98KKC< =FI D< $K >F<J N@K?FLK J8P@E> K?8K $cM< 8CeI<8;P I<8; I<8DJ 8E; I<8DJ F= 8;M@:< 8E; K?< DFJK ?<CG $ >FK =IFD@K N8J I<8C@Q@E> K?8K 8 J@Q89C< =I8:K@FE FK?<I :I<8K@M< GIF=<JJ@FE8CJ

?8; K?< J8D< GIF9C<D 8E; :FLC;EcK 9<8K @K <@K?<I EF D8KK<I ?FN I<8JFE89C< 8CCK?<8;M@:<JFLE;J

`2?8K ;F PFL ;F N?<E PFL :8EcK NFIBa DP =I@<E;J 8JB<; D<hFEM<IJ8K@FEGIF989CPEFK8::LI8K<K?@J@J8M<IPCFFJ<>@JKi

 E; $ I<GC@<; K?8K $ LJL8CCP 9IFNJ< I8E;FD N<9J@K<J FI N8K:? 8 J?FIKM@;<F

`2<CCa K?<P J8@; `@= PFL BEFN PFL :8EcK NFIB =FI 8 N?@C< PFLJ?FLC;N8K:?8DFM@<FIJFD<K?@E>a

`0E=FIKLE8K<CPa $ I<GC@<; `$ ?8M< KF ;F JFD<K?@E> N?FJ< K@D<:FD<J @E J?FIK LE@KJ C@B< 9IFNJ@E> K?< 2<9 FI N8K:?@E> J?FIK

 M@;<FJ 9<:8LJ< $ D@>?K 9<:FD< 89C< KF NFIB 8>8@E 8K 8EP K@D< 8E;$:8EcKGI<;@:KN?<Ega

 E;K?<E$JKFGG<;9<:8LJ<$c;ALJK?8;8I<M<C8K@FE

$c; 8CN8PJ K?FL>?K F= DP NFIB:P:C< 8J JFD<K?@E> chaotic, JFD<eK?@E> unpredictable. $ E<M<I LJ<; K?FJ< NFI;J 9LK K?8K N8J K?< N8P $ treated @K

LK ?<I< DP =I@<E;J J<<D<; KF 9< @DGCP@E>gN?8K 8 JKI8E><

K?FL>?KgK?8K other G<FGC< :FLC; GI<;@:K N?<E K?<P NFLC; 9<:FD<89C<KFNFIB8>8@E8E;JKIL:KLI<K?<@IK@D<8::FI;@E>CP

 E; @K F::LII<; KF D< =FI K?< =@IJK K@D< K?8K $ D@>?K ?8M< 9<<E:FDD@KK@E> K?8K ;8DE<; FC; :?<JKELK K?< (@E; +IFA<:K@FE !8CC8e:P  I@>?K FLK K?<I< @E DP FI;@E8IP <M<IP;8P C@=< @EJK<8; F= ?@>?89JKI8:K@FE

http://lesswrong.com/lw/wb/chaotic_inversion/ 

http://lesswrong.com/lw/un/on_doing_the_impossible/  +8><b(@E;+IFA<:K@FE!8CC8:Pc

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 188/309

(8P9< @K N8JEcK K?8K DP GIF;L:K@M@KP N8J unusually chaotic;D8P9<$N8JALJK unusually stupid N@K?I<JG<:KKFGI<;@:K@E>@K

/?8KcJ N?8K @EM<IK<; JKLG@;@KP CFFBJ C@B<g:?8FJ .FD<K?@E> 

?8I; KF ?8E;C< ?8I; KF >I8JG ?8I; KF >L<JJ JFD<K?@E> PFL :8EcK ;F8EPK?@E> N@K? $KcJ EFK ALJK 8E @;@FD =FI ?@>? 89JKI8:K K?@E>J C@B<

 IK@=@:@8C$EK<CC@><E:< $K:8E8GGCP@EFI;@E8IPC@=<KFF

 E; K?< I<8JFE N< ;FEcK K?@EB F= K?< 8CK<IE8K@M< <OGC8E8K@FE`$cD JKLG@;a @J not  g$ JLJG<:KgK?8K N< K?@EB JF ?@>?CP F= FLIeJ<CM<J $KcJ ALJK K?8K N< ;FEcK K?@EB F= FLIJ<CM<J 8K 8CC 2< ALJK J<<8:?8FK@:=<8KLI<F=K?<<EM@IFED<EK

.F EFN @KcJ F::LII<; KF D< K?8K DP GIF;L:K@M@KP GIF9C<D D8P EFK9<:?8FJ9LKDPFNEJKLG@;@KP

 E; K?8K D8P FI D8P EFK ?<CG 8EPK?@E> $K :<IK8@ECP ;F<JEcK =@O K?< GIF9C<D I@>?K 8N8P .8P@E> `$cD @>EFI8EKa ;F<JEcK D8B< PFLBEFNC<;><89C<

LK@K@J8KC<8JK8;@==<I<EKG8K?K?8EJ8P@E> @KcJKFF:?8FK@:a

+8><b(@E;+IFA<:K@FE!8CC8:Pc

%1+,*!'-+ &+/*+,'%1+,*!'-+)-+,!'&+

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 189/309

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 190/309

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 191/309

1. The Parable of the Dagger

*E:<LGFE8K@D<K?<I<N8J8:FLIKA<JK<IN?F;899C<;@ECF>@:

/?< A<JK<I GI<J<EK<; K?< B@E> N@K? KNF 9FO<J 0GFE K?< =@IJK9FON8J@EJ:I@9<;

`@K?<IK?@J9FO:FEK8@EJ8E8E>IP=IF>FIK?<9FON@K?8 =8CJ<@EJ:I@GK@FE:FEK8@EJ8E8E>IP=IF>9LKEFK9FK?a

*EK?<J<:FE;9FON8J@EJ:I@9<;

`@K?<IK?@J9FO:FEK8@EJ>FC;8E;K?<9FON@K?8=8CJ<

@EJ:I@GK@FE:FEK8@EJ8E8E>IP=IF>FIK?@J9FO:FEK8@EJ8E8E>IP=IF>8E;K?<9FON@K?8KIL<@EJ:I@GK@FE:FEK8@EJ

 >FC;a

 E; K?< A<JK<I J8@; KF K?< B@E> `*E< 9FO :FEK8@EJ 8E 8E>IP =IF> K?< FK?<I 9FO >FC; 8E; FE< 8E; FECP FE< F= K?< @EJ:I@GK@FEJ@JKIL<a

/?< B@E> FG<E<; K?< NIFE> 9FO 8E; N8J J8M8><; 9P 8E 8E>IP 

=IF>4FL J<<a K?< A<JK<I J8@; `C<K LJ ?PGFK?<J@Q< K?8K K?< =@IJK @EeJ:I@GK@FE @J K?< KIL< FE< /?<E JLGGFJ< K?< =@IJK 9FO :FEK8@EJ >FC;/?<E K?< FK?<I 9FO NFLC; ?8M< 8E 8E>IP =IF> N?@C< K?< 9FO N@K? 8 KIL< @EJ:I@GK@FE NFLC; :FEK8@E >FC; N?@:? NFLC; D8B< K?< J<:FE;JK8K<D<EK KIL< 8J N<CC )FN ?PGFK?<J@Q< K?8K K?< =@IJK @EJ:I@GK@FE@J =8CJ< 8E; K?8K K?< =@IJK 9FO :FEK8@EJ >FC; /?<E K?< J<:FE; @EeJ:I@GK@FENFLC;9<ga

/?<B@E>FI;<I<;K?<A<JK<IK?IFNE@EK?<;LE><FEJ

  ;8P C8K<I K?< A<JK<I N8J 9IFL>?K 9<=FI< K?< B@E> @E :?8@EJ 8E;J?FNEKNF9FO<J

`*E< 9FO :FEK8@EJ 8 B<Pa J8@; K?< B@E> `KF LECF:B PFLI :?8@EJ8E; @= PFL =@E; K?< B<P PFL 8I< =I<< LK K?< FK?<I 9FO :FEK8@EJ 8 ;8>><I=FIPFLI?<8IK@=PFL=8@Ca

 E;K?<=@IJK9FON8J@EJ:I@9<;

http://lesswrong.com/lw/ne/the_parable_of_the_dagger/ 

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 192/309

`@K?<I9FK?@EJ:I@GK@FEJ8I<KIL<FI9FK?@EJ:I@GK@FEJ8I<=8CJ<a

 E;K?<J<:FE;9FON8J@EJ:I@9<;

`/?@J9FO:FEK8@EJK?<B<Pa

/?< A<JK<I I<8JFE<; K?LJCP `.LGGFJ< K?< =@IJK @EJ:I@GK@FE @JKIL< /?<E K?< J<:FE; @EJ:I@GK@FE DLJK 8CJF 9< KIL< )FN JLGGFJ<K?< =@IJK @EJ:I@GK@FE @J =8CJ< /?<E 8>8@E K?< J<:FE; @EJ:I@GK@FEDLJK 9< KIL< .F K?< J<:FE; 9FO DLJK :FEK8@E K?< B<P @= K?< =@IJK@EJ:I@GK@FE @J KIL< 8E; 8CJF @= K?< =@IJK @EJ:I@GK@FE @J =8CJ< /?<I<e

=FI<K?<J<:FE;9FODLJKCF>@:8CCP:FEK8@EK?<B<Pa/?<A<JK<IFG<E<;K?<J<:FE;9FO8E;=FLE;8;8>><I

`#FNa :I@<; K?< A<JK<I @E ?FIIFI 8J ?< N8J ;I8>><; 8N8P `$KcJCF>@:8CCP@DGFJJ@9C<a

`$K @J <EK@I<CP GFJJ@9C<a I<GC@<; K?< B@E> `$ D<I<CP NIFK< K?FJ<@EJ:I@GK@FEJ FE KNF 9FO<J 8E; K?<E $ GLK K?< ;8>><I @E K?< J<:FE;FE<a

(Adapted from Raymond Smullyan.)

-%&6+-!,'/'*+

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 193/309

2. The Parable of Hemlock 

Followup to /?<+8I89C<F=K?<8>><I

`CCD<E8I<DFIK8C .F:I8K<J@J8D8E /?<I<=FI<.F:I8K<J@JDFIK8Ca

 gI@JKFKC<hi

Socrates raised the glass of hemlock to his lips…`F PFL JLGGFJ<a 8JB<; FE< F= K?< FECFFB<IJ `K?8K <M<E ?<De

CF:BN@CCEFK9<<EFL>?KFB@CCJFN@J<8E;>FF;8D8Ea`)Fa I<GC@<; 8EFK?<I 9PJK8E;<I 8 JKL;<EK F= G?@CFJFG?P `8CC

D<E 8I< DFIK8C 8E; .F:I8K<J @J 8 D8E 8E; @= 8 DFIK8C ;I@EB ?<DeCF:BJLI<CP?<;@<Ja

`2<CCa J8@; K?< FECFFB<I `N?8K @= @K ?8GG<EJ K?8K .F:I8K<J isn’t DFIK8Ca

`)FEJ<EJ<a I<GC@<; K?< JKL;<EK 8 C@KKC< J?8IGCP `8CC D<E 8I< DFIeK8C by definition @K @J G8IK F= N?8K N< D<8E 9P K?< NFI; bD8Ec CCD<E 8I< DFIK8C .F:I8K<J @J 8 D8E K?<I<=FI< .F:I8K<J @J DFIK8C $K@JEFKD<I<CP8>L<JJ9LK8 logical certainty.a

`$ JLGGFJ< K?8KcJ I@>?K_a J8@; K?< FECFFB<I `*? CFFB .F:I8K<J8CI<8;P;I8EBK?<?<DCF:BN?@C<N<N<I<K8CB@E>a

`4<J ?< J?FLC; 9< B<<C@E> FM<I 8EP D@ELK< EFNa J8@; K?< JKLe;<EK

 And they waited, and they waited, and they waited…`.F:I8K<J8GG<8IJEFKKF9<DFIK8CaJ8@;K?<FECFFB<I`/?<E .F:I8K<J DLJK EFK 9< 8 D8Ea I<GC@<; K?< JKL;<EK `CC

D<E 8I< DFIK8C .F:I8K<J @J EFK DFIK8C K?<I<=FI< .F:I8K<J @J EFK 8 D8E E;K?8K@JEFKD<I<CP8>L<JJ9LK8 logical certainty.a

/?< =LE;8D<EK8C GIF9C<D N@K? 8I>L@E> K?8K K?@E>J 8I< KIL< `9P ;<=@E@K@FEa @J K?8K PFL :8EcK D8B< I<8C@KP >F 8 ;@==<I<EK N8P 9P :?FFJ@E>8;@==<I<EK;<=@E@K@FE

4FL :FLC; I<8JFE G<I?8GJ 8J =FCCFNJ `CC K?@E>J $ ?8M< F9eJ<IM<; N?@:? N<8I :CFK?@E> JG<8B C8E>L8>< 8E; LJ< KFFCJ ?8M< 8CJFJ?8I<; :<IK8@E FK?<I GIFG<IK@<J 8J N<CC JL:? 8J 9I<8K?@E> 8@I 8E;GLDG@E> I<; 9CFF; /?< C8JK K?@IKP b?LD8EJc 9<CFE>@E> KF K?@J :CLJe

http://lesswrong.com/lw/nf/the_parable_of_hemlock/ 

+8><b/?<+8I89C<F=K?<8>><Ic http://machall.com/view.php?date=2003-04-21

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 194/309

K<I N?FD $ F9J<IM<; KF ;I@EB ?<DCF:B JFFE =<CC FM<I 8E; JKFGG<;DFM@E> .F:I8K<J N<8IJ 8 KF>8 JG<8BJ =CL<EK 8E:@<EK "I<<B 8E;;I8EB ?<DCF:B =IFD 8 :LG .F $ GI<;@:K K?8K .F:I8K<J N@CC B<<C FM<I

@EK?<E<OK=@M<D@ELK<JaLK K?8K NFLC; 9< D<I< guessing. $K NFLC;EcK 9< PcBEFN 89JFe

CLK<CP 8E; <K<IE8CCP :<IK8@E /?< "I<<B G?@CFJFG?<IJgC@B< DFJKGI<J:@<EK@=@:G?@CFJFG?<IJgN<I<I8K?<I=FE;F=:<IK8@EKP

'L:B@CP K?< "I<<B G?@CFJFG?<IJ ?8M< 8 :ILJ?@E> I<AF@E;<I KF PFLI HL<JK@FE@E> 4FL ?8M< D@JLE;<IJKFF; K?< D<8E@E> F= `CC ?LeD8EJ 8I< DFIK8Ca K?<P J8P $K @J EFK 8 D<I< observation. $K @J G8IKF= K?< definition F= K?< NFI; `?LD8Ea (FIK8C@KP @J FE< F= J<M<I8CGIFG<IK@<J K?8K 8I< @E;@M@;L8CCP E<:<JJ8IP 8E; KF><K?<I JL==@:@<EKKF ;<K<ID@E< D<D9<IJ?@G @E K?< :C8JJ ?LD8Ea /?< JK8K<D<EK `CC?LD8EJ 8I< DFIK8Ca @J 8 CF>@:8CCP M8C@; KILK? 89JFCLK<CP LEHL<JK@FEe89C< E; @= .F:I8K<J @J ?LD8E ?< must  9< DFIK8C @K @J 8 CF>@:8C;<;L:K@FE8J:<IK8@E8J:<IK8@E:8E9<

LK K?<E N< :8E E<M<I BEFN =FI :<IK8@E K?8K .F:I8K<J @J 8 `?LeD8Ea LEK@C 8=K<I .F:I8K<J ?8J 9<<E F9J<IM<; KF 9< DFIK8C $K ;F<JEF >FF; KF F9J<IM< K?8K .F:I8K<J JG<8BJ =CL<EK "I<<B FI K?8K.F:I8K<J ?8J I<; 9CFF; FI <M<E K?8K .F:I8K<J ?8J ?LD8E )

)FE< F= K?<J< :?8I8:K<I@JK@:J 8I< logically equivalent  KF DFIK8C@KP4FL ?8M< KF see him die 9<=FI< PFL :8E :FE:CL;< K?8K ?< N8J ?LD8E

hE; <M<E K?<E @KcJ EFK @E=@E@K<CP :<IK8@E 2?8K @= .F:I8K<JI@J<J =IFD K?< >I8M< 8 E@>?K 8=K<I PFL J<< ?@D ;@< *I DFI< I<8Ce@JK@:8CCP N?8K @= .F:I8K<J @J J@>E<; LG =FI :IPFE@:J $= DFIK8C@KP @J;<=@E<; KF D<8E =@E@K< C@=<JG8E K?<E PFL :8E E<M<I I<8CCP  know @= JFD<FE< N8J ?LD8E LEK@C PFLcM< F9J<IM<; KF K?< <E; F= <K<IE@eKPgALJK KF D8B< JLI< K?<P ;FEcK :FD< 98:B *I PFL :FLC; think PFLJ8N .F:I8K<J B<<C FM<I 9LK @K :FLC; 9< 8E @CCLJ@FE GIFA<:K<; FEKF

 PFLI <P<J N@K? 8 I<K@E8C J:8EE<I *I D8P9< PFL ALJK ?8CCL:@E8K<; K?< N?FC<K?@E>_i

/?< GIF9C<D N@K? JPCCF>@JDJ @J K?8K K?<PcI< always M8C@; `CC?LD8EJ 8I< DFIK8C .F:I8K<J @J ?LD8E K?<I<=FI< .F:I8K<J @J DFIK8Ca@Jg@= PFL KI<8K @K 8J 8 CF>@:8C JPCCF>@JDgCF>@:8CCP M8C@; N@K?@E FLIFNE LE@M<IJ< $KcJ 8CJF CF>@:8CCP M8C@; N@K?@E E<@>?9FI@E> M<I<KK9I8E:?<J @E N?@:? ;L< KF 8 JC@>?KCP ;@==<I<EK <MFCM<; 9@F:?<D@JKIP

+8><b9JFCLK<LK?FI@KPc +8><b$E=@E@K<<IK8@EKPc

-%&6+-!,'/'*+

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 195/309

?<DCF:B @J 8 ;<C@:@FLJ KI<8K I8K?<I K?8E 8 GF@JFE E; @KcJ CF>@:8CCP  M8C@; <M<E @E LE@M<IJ<J N?<I< .F:I8K<J E<M<I <O@JK<; FI =FI K?8KD8KK<IN?<I<?LD8EJE<M<I<O@JK<;

/?< 8P<J@8E ;<=@E@K@FE F= <M@;<E:< =8MFI@E> 8 ?PGFK?<J@J @J <Me@;<E:< N?@:? N< 8I< DFI< C@B<CP KF J<< @= K?< ?PGFK?<J@J @J KIL< K?8E@= @K @J =8CJ< *9J<IM@E> K?8K 8 JPCCF>@JD @J CF>@:8CCP M8C@; :8E E<M<I9< <M@;<E:< =8MFI@E> 8EP <DG@I@:8C GIFGFJ@K@FE 9<:8LJ< K?< JPCCFe

 >@JDN@CC9<CF>@:8CCPM8C@;N?<K?<IK?8KGIFGFJ@K@FE@JKIL<FI=8CJ<

.PCCF>@JDJ 8I< M8C@; @E 8CC GFJJ@9C< NFIC;J 8E; K?<I<=FI< F9J<IMe@E> K?<@I M8C@;@KP E<M<I K<CCJ LJ 8EPK?@E> 89FLK which GFJJ@9C< NFIC;

 N<8:KL8CCPC@M<@E

/?@J ;F<JEcK D<8E K?8K CF>@: @J LJ<C<JJgALJK K?8K CF>@: :8E FECP K<CC LJ K?8K N?@:? in some sense, N< 8CI<8;P BEFN LK N< ;F EFK 8Ce

 N8PJ 9<C@<M< N?8K N< BEFN $J K?< ELD9<I GI@D< P  M@IKL< F= ?FN $ ;<=@E< DP ;<:@D8C JPJK<D 8E; DP 8O@FDJ F= 8I@K?eD<K@: $ ?8M< 8CI<8;P ;<K<ID@E<; DP 8EJN<I KF K?@J HL<JK@FEg9LK$ ;F EFK BEFN N?8K DP 8EJN<I @J P<K 8E; $ DLJK ;F JFD< CF>@: KF=@E;FLK

.@D@C8ICP @= $ =FID K?< LE:<IK8@E <DG@I@:8C ><E<I8C@Q8K@FE `#LeD8EJ 8I< MLCE<I89C< KF ?<DCF:Ba 8E; K?< LE:<IK8@E <DG@I@:8C >L<JJ

`.F:I8K<J @J ?LD8Ea CF>@: :8E K<CC D< K?8K DP GI<M@FLJ >L<JJ<J 8I<GI<;@:K@E>K?8K.F:I8K<JN@CC9<MLCE<I89C<KF?<DCF:B

$KcJ 9<<E JL>><JK<; K?8K N< :8E M@<N CF>@:8C I<8JFE@E> 8J I<JFCMe@E> FLI LE:<IK8@EKP 89FLK @DGFJJ@9C< GFJJ@9C< NFIC;Jg<C@D@E8K@E> GIF989@C@KP D8JJ @E CF>@:8CCP @DGFJJ@9C< NFIC;J N?@:? N< ;@; EFKBEFN KF 9< CF>@:8CCP @DGFJJ@9C< $E K?@J J<EJ< CF>@:8C 8I>LD<EK :8E9< KI<8K<;8JF9J<IM8K@FE

LK N?<E PFL K8CB 89FLK 8E <DG@I@:8C GI<;@:K@FE C@B< `.F:I8K<J

@J >F@E> KF B<<C FM<I 8E; JKFG 9I<8K?@E>a FI `.F:I8K<J @J >F@E> KF ;F=@=KP ALDG@E> A8:BJ 8E; K?<E :FDG<K< @E K?< *CPDG@:J E<OK P<8IaK?8K@J8D8KK<IF=GFJJ@9C<NFIC;JEFK@DGFJJ@9C<GFJJ@9C<NFIC;J

'F>@: :8E K<CC LJ N?@:? ?PGFK?<J<J D8K:? LG KF N?@:? F9J<Ie M8K@FEJ 8E; @K :8E K<CC LJ N?8K K?<J< ?PGFK?<J<J GI<;@:K =FI K?<=LKLI<g@K :8E 9I@E> FC; F9J<IM8K@FEJ 8E; GI<M@FLJ >L<JJ<J KF 9<8IFE 8 E<N GIF9C<D LK CF>@: E<M<I =C8KCP J8PJ `.F:I8K<J will  JKFG9I<8K?@E> EFNa 'F>@: E<M<I ;@:K8K<J 8EP <DG@I@:8C HL<JK@FE @K E<Me

+8><bE$EKL@K@M<OGC8E8K@FEF=8P<J/?<FI<Dc +8><b#FNKFFEM@E:<(</?8Kc

,(*$'%$'# 

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 196/309

<I J<KKC<J 8EP I<8CeNFIC; HL<IP N?@:? :FLC; 9P 8EP JKI<K:? F= K?<@D8>@E8K@FE>F<@K?<IN8P

 %LJKI<D<D9<IK?<'@K8EP>8@EJK'F>@:

'F>@:JK8PJKIL<N?<I<M<IPFLD8P>F.FCF>@:E<M<IK<CCJPFLN?<I<PFLC@M<

-%&6+-!,'/'*+

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 197/309

3. Words as Hidden Inferences

Followup to /?<+8I89C<F=#<DCF:B

.LGGFJ< $ =@E; 8 98II<C J<8C<; 8K K?< KFG 9LK N@K? 8 ?FC< C8I><<EFL>? =FI 8 ?8E; $ I<8:? @E 8E; =<<C 8 JD8CC :LIM<; F9A<:K $GLCC K?< F9A<:K FLK 8E; @KcJ 9CL<g8 9CL@J? <>> )<OK $ I<8:? @E 8E;=<<C JFD<K?@E> ?8I; 8E; =C8K N@K? <;><JgN?@:? N?<E $ <OKI8:K @KGIFM<J KF 9< 8 I<; :L9< $ GLCC FLK <>>J 8E; :L9<J 8E; <M<IP <>>@J9CL<8E;<M<IP:L9<@JI<;

)FN $ I<8:? @E 8E; $ =<<C 8EFK?<I <>>eJ?8G<; F9A<:K <=FI< $GLCC@KFLK8E;CFFB$?8M<KF>L<JJ 2?8KN@CC@KCFFBC@B<

/?< <M@;<E:< ;F<JEcK GIFM< K?8K <M<IP <>> @E K?< 98II<C @J 9CL<8E; <M<IP :L9< @J I<; /?< <M@;<E:< ;F<JEcK <M<E 8I>L< K?@J 8CC K?8KJKIFE>CP @J EFK 8 C8I>< J8DGC< J@Q< )FE<K?<C<JJ $cCC >L<JJ K?8KK?@J <>>eJ?8G<; F9A<:K @J 9CL<gFI 8J 8 ILEE<IeLG >L<JJ I<; $= $

 >L<JJ 8EPK?@E> <CJ< K?<I<cJ 8J D8EP GFJJ@9@C@K@<J 8J ;@JK@E>L@J?89C<:FCFIJg8E; =FI K?8K D8KK<I N?F J8PJ K?< <>> ?8J KF 9< 8 J@E>C<J?8;< (8P9<@K?8J8G@:KLI<F=8?FIJ<G8@EK<;FE

.F $ J8P `9CL<a N@K? 8 ;LK@=LC G8K@E8 F= ?LD@C@KP !FI $ 8D8 JFG?@JK@:8K<; I8K@FE8C@JKeKPG< G<IJFE 8E; $ B<<G KI8:B F= DP 

8JJLDGK@FEJ 8E; ;<G<E;<E:@<Jg$ >L<JJ 9LK $cD 8N8I< K?8K $cD >L<JJ@E>_I@>?K

LK N?<E 8 C8I>< P<CCFN JKI@G<; =<C@E<eJ?8G<; F9A<:K C<8GJ FLK 8KD< =IFD K?< J?8;FNJ $ K?@EB `4@B<J K@><Ia )FK `#D_ F9eA<:KJ N@K? K?< GIFG<IK@<J F= C8I><E<JJ P<CCFNE<JJ JKI@G<;E<JJ 8E;=<C@E< J?8G< ?8M< GI<M@FLJCP F=K<E GFJJ<JJ<; K?< GIFG<IK@<J b?LEe

 >IPc 8E; b;8E><IFLJc 8E; K?LJ 8CK?FL>? @K @J EFK CF>@:8CCP E<:<JJ8IP@K D8P 9< 8E <DG@I@:8CCP >FF; >L<JJ K?8K aaauuughhhh -0)#

-0)#"0'+a/?< ?LD8E 9I8@E =FI JFD< F;; I<8JFE J<<DJ KF ?8M< 9<<E8;8GK<; KF D8B< K?@J @E=<I<E:< HL@:BCP 8LKFD8K@:8CCP 8E; N@K?FLKB<<G@E><OGC@:@KKI8:BF=@KJ8JJLDGK@FEJ

 E; @= $ E8D< K?< <>>eJ?8G<; F9A<:KJ `9C<>>Ja h=FI 9CL< <>>Ji8E; K?< I<; :L9<J `IL9<Ja K?<E N?<E $ I<8:? @E 8E; =<<C 8EFK?<I<>>eJ?8G<; F9A<:K $ D8P K?@EB Oh, it’s a blegg, I8K?<I K?8E :FEJ@;e<I@E>8CCK?8KGIF9C<DeF=e@E;L:K@FEJKL==

http://lesswrong.com/lw/ng/words_as_hidden_inferences/  +8><b/?<+8I89C<F=#<DCF:Bc

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 198/309

$K @J 8 :FDDFE D@J:FE:<GK@FE K?8K PFL :8E ;<=@E< 8 NFI; 8EP  N8PPFLC@B<

/?@J NFLC; 9< KIL< if  K?< 9I8@E KI<8K<; NFI;J 8J GLI<CP CF>@:8C

:FEJKIL:KJ I@JKFK<C@8E :C8JJ<J 8E; PFL E<M<I KFFB FLK 8EP DFI<@E=FID8K@FEK?8EPFLGLK@E

4<K K?< 9I8@E >F<J FE 89FLK @KJ NFIB F= :8K<>FI@Q8K@FE N?<K?<IFI EFK N< :FEJ:@FLJCP 8GGIFM< `CC ?LD8EJ 8I< DFIK8C .F:I8K<J@J 8 ?LD8E K?<I<=FI< .F:I8K<J @J DFIK8CagK?LJ JG8B< K?< 8E:@<EK"I<<B G?@CFJFG?<IJ 2<CC @= DFIK8C@KP @J G8IK F= PFLI CF>@:8C ;<=e@E@K@FE F= `?LD8Ea PFL :8EcK CF>@:8CCP :C8JJ@=P .F:I8K<J 8J ?LD8ELEK@C PFL F9J<IM< ?@D KF 9< DFIK8C LKgK?@J @J K?< GIF9eC<DgI@JKFKC< BE<N G<I=<:KCP N<CC K?8K .F:I8K<J N8J 8 ?LD8E

 I@JKFKC<cJ 9I8@E GC8:<; .F:I8K<J @E K?< `?LD8Ea :8K<>FIP 8J <==@e:@<EKCP 8J PFLI FNE 9I8@E :8K<>FI@Q<J K@><IJ 8GGC<J 8E; <M<IPK?@E> <CJ< @E @KJ <EM@IFED<EK .N@=KCP J@C<EKCP 8E; N@K?FLK :FEJ:@FLJ 8GeGIFM8C

 I@JKFKC< C8@; ;FNE ILC<J LE;<I N?@:? EF FE< :FLC; :FE:CL;<.F:I8K<J N8J `?LD8Ea LEK@C 8=K<I ?< ;@<; )FE<K?<C<JJ I@JKFKC<8E; ?@J JKL;<EKJ N<EK FE :FE:CL;@E> K?8K C@M@E> G<FGC< N<I< ?LeD8EJ 8E; K?<I<=FI< DFIK8C K?<P J8N ;@JK@E>L@J?@E> GIFG<IK@<J JL:?

8J ?LD8E =8:<J 8E; ?LD8E 9F;@<J 8E; K?<@I 9I8@EJ D8;< K?< C<8GKF@E=<II<;GIFG<IK@<JJL:?8JDFIK8C@KP

(@JLE;<IJK8E;@E> K?< NFIB@E> F= PFLI FNE D@E; ;F<J not,K?8EB=LCCP GI<M<EK K?< D@E; =IFD ;F@E> @KJ NFIB *K?<IN@J< I@JeKFK<C@8EJ NFLC; ?8M< JK8IM<; LE89C< KF :FE:CL;< K?8K 8E F9A<:K N8J<;@9C<D<I<CP9<:8LJ<@KCFFB<;8E;=<CKC@B<898E8E8

.F K?< I@JKFK<C@8EJ N<EK FE :C8JJ@=P@E> <EM@IFED<EK8C F9A<:KJFE K?< 98J@J F= G8IK@8C @E=FID8K@FE K?< N8P G<FGC< ?8; 8CN8PJ;FE< .KL;<EKJ F= I@JKFK<C@8E CF>@: N<EK FE K?@EB@E> <O8:KCP K?<J8D< N8P 9LK K?<P ?8; 8:HL@I<; 8E <IIFE<FLJ G@:KLI< F= what K?<P 

 N<I<;F@E>

$= PFL 8JB<; 8E I@JKFK<C@8E G?@CFJFG?<I N?<K?<I 8IFC K?< >IFe:<I N8J DFIK8C K?<P NFLC; J8P `4<Ja $= PFL 8JB<; K?<D ?FN K?<P BE<N K?<P NFLC; J8P `CC ?LD8EJ 8I< DFIK8C 8IFC @J ?LD8EK?<I<=FI< 8IFC @J DFIK8Ca JB K?<D N?<K?<I @K N8J 8 >L<JJ FI 8 :<IK8@EKP 8E; K?<P NFLC; J8P @K N8J 8 :<IK8@EKP h@= PFL 8JB<; 9<=FI<

+8><b/?<+8I89C<F=#<DCF:Bc +8><b/?<+8I89C<F=#<DCF:Bc

-%&6+-!,'/'*+

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 199/309

K?< J@OK<<EK? :<EKLIP 8K C<8JKi JB K?<D ?FN K?<P BE<N K?8K?LD8EJ N<I< DFIK8C 8E; K?<P NFLC; J8P @K N8J <JK89C@J?<; 9P ;<=@eE@K@FE

/?< I@JKFK<C@8EJ N<I< JK@CC K?< J8D< G<FGC< K?<P I<K8@E<; K?<@IFI@>@E8C E8KLI<J 9LK K?<P ?8; 8:HL@I<; @E:FII<:K 9<C@<=J 89FLK K?<@IFNE =LE:K@FE@E>  /?<P CFFB<; @EKF K?< D@IIFI F= J<C=e8N8I<E<JJ8E; J8N JFD<K?@E> LEC@B< K?<@I KIL< J<CM<J K?<P I<=C<:K<; @E:FII<:KeCP

4FLI 9I8@E ;F<JEcK KI<8K NFI;J 8J CF>@:8C ;<=@E@K@FEJ N@K? EF<DG@I@:8C :FEJ<HL<E:<J 8E; JF E<@K?<I J?FLC; PFL /?< D<I< 8:KF= :I<8K@E> 8 NFI; :8E :8LJ< PFLI D@E; KF 8CCF:8K< 8 :8K<>FIP 8E;K?<I<9P KI@>><I LE:FEJ:@FLJ @E=<I<E:<J F= J@D@C8I@KP *I 9CF:B @Ee=<I<E:<J F= J@D@C8I@KP @= $ :I<8K< KNF C89<CJ $ :8E ><K PFLI D@E; KF8CCF:8K< KNF :8K<>FI@<J )FK@:< ?FN $ J8@; `PFLa 8E; `PFLI 9I8@Ea8J@=K?<PN<I<;@==<I<EKK?@E>J

(8B@E> <IIFIJ 89FLK K?< @EJ@;< F= PFLI ?<8; ;F<JEcK :?8E>< N?8KcJ K?<I< FK?<IN@J< I@JKFKC< NFLC; ?8M< ;@<; N?<E ?< :FE:CL;e<; K?8K K?< 9I8@E N8J 8E FI>8E =FI :FFC@E> K?< 9CFF; +?@CFJFG?@:8CD@JK8B<J LJL8CCP ;FEcK @EK<I=<I< N@K? 9C@EBeF=e8Ee<P< G<I:<GKL8C @Ee=<I<E:<J

LK G?@CFJFG?@:8C D@JK8B<J :8E J<M<I<CP D<JJ LG K?< ;<C@9<I8K<K?@EB@E> GIF:<JJ<J K?8K N< LJ< KF KIP KF :FII<:K FLI =@IJK @DGI<JeJ@FEJ $= PFL 9<C@<M< K?8K PFL :8E `;<=@E< 8 NFI; 8EP N8P PFL C@B<a

 N@K?FLK I<8C@Q@E> K?8K PFLI 9I8@E >F<J FE :8K<>FI@Q@E> N@K?FLK PFLI:FEJ:@FLJ FM<IJ@>?K K?<E PFL NFEcK K8B< K?< <==FIK KF :?FFJ< PFLI;<=@E@K@FEJN@J<CP

+8><b/?<'<EJ/?8K.<<J$KJ!C8NJc http://lesswrong.com/lw/mg/the_twoparty_swindle/ 

/'*++!&! &*&+

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 200/309

4. Extensions and Intensions

Followup to 2FI;J8J#@;;<E$E=<I<E:<J

`2?8K@JI<;a`-<;@J8:FCFIa`2?8KcJ8:FCFIa`:FCFI@J8GIFG<IKPF=8K?@E>a

LK N?8K @J 8 K?@E> E; N?8KcJ 8 GIFG<IKP .FFE K?< KNF 8I<CFJK @E 8 D8Q< F= NFI;J ;<=@E<; @E FK?<I NFI;J K?< GIF9C<D K?8K.K<M<E #8IE8; FE:< ;<J:I@9<; 8J KIP@E> KF C<8IE ?@E<J< =IFD 8 ?@E<J<?@E<J<;@:K@FE8IP

 CK<IE8K@M<CP @= PFL 8JB<; D< `2?8K @J I<;a $ :FLC; GF@EK KF8 JKFG J@>E K?<E KF JFD<FE< N<8I@E> 8 I<; J?@IK 8E; 8 KI8==@: C@>?KK?8K ?8GG<EJ KF 9< I<; 8E; 9CFF; =IFD N?<I< $ 8::@;<EK8CCP :LKDPJ<C= 8E; 8 I<; 9LJ@E<JJ :8I; 8E; K?<E $ :FLC; :8CC LG 8 :FCFI

 N?<<C FE DP :FDGLK<I 8E; DFM< K?< :LIJFI KF K?< I<; 8I<8 /?@J NFLC; GIF989CP 9< JL==@:@<EK K?FL>? @= PFL BEFN N?8K K?< NFI;`)Fa D<8EJ K?< KILCP JKI@:K  NFLC; @EJ@JK K?8K $ GF@EK KF K?< JBP 

8E;J8P`)Fa$ K?@EB $ JKFC< K?@J <O8DGC< =IFD . $ #8P8B8N8gK?FL>? $cD

I<8CCP EFK JLI< 9<:8LJ< $ ?<8I; K?@J N8P 98:B @E K?< @E;@JK@E:K 9CLIF= DP :?@C;?FF; h2?<E $ N8J DP =8K?<I 8::@;<EK8CCP ;<C<K<; 8CCDP:FDGLK<I=@C<J $?8M<EFD<DFIPF=8EPK?@E>9<=FI<K?8Ki

LK K?8KcJ ?FN $ I<D<D9<I =@IJK C<8IE@E> 89FLK K?< ;@==<I<E:<9<KN<<E @EK<EJ@FE8C 8E; <OK<EJ@FE8C ;<=@E@K@FE /F >@M< 8E `@EK<EeJ@FE8C ;<=@E@K@FEa @J KF ;<=@E< 8 NFI; FI G?I8J< @E K<IDJ F= FK?<I

 NFI;J 8J 8 ;@:K@FE8IP ;F<J /F >@M< 8E `<OK<EJ@FE8C ;<=@E@K@FEa @JKF GF@EK KF <O8DGC<J 8J 8;LCKJ ;F N?<E K<8:?@E> :?@C;I<E /?<GI<:<;@E> J<EK<E:< >@M<J 8E @EK<EJ@FE8C ;<=@E@K@FE F= `<OK<EJ@FE8C;<=@E@K@FEa N?@:? D8B<J @K 8E <OK<EJ@FE8C <O8DGC< F= `@EK<EJ@FE8C;<=@E@K@FEa

http://lesswrong.com/lw/nh/extensions_and_intensions/  +8><b2FI;J8J#@;;<E$E=<I<E:<Jc

http://www.ecs.soton.ac.uk/%7Eharnad/Papers/Harnad/harnad90.sgproblem.html  +8><b+FJ@K@M<@8J'FFB$EKFK?<8IBc

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 201/309

$E #FCCPNFF; -8K@FE8C@KP 8E; GFGLC8I :LCKLI< ><E<I8CCP `I8K@FeE8C@JKJa 8I< ;<G@:K<; 8J NFI;eF9J<JJ<; =CF8K@E> @E <E;C<JJ M<I98CJG8:<;@J:FEE<:K<;=IFDI<8C@KP

LK K?< 8:KL8C /I8;@K@FE8C -8K@FE8C@JKJ ?8M< CFE> @EJ@JK<; FED8@EK8@E@E>8K@>?K:FEE<:K@FEKF<OG<I@<E:<

`$=PFLCFFB@EKF8K<OK9FFBF=:?<D@JKIP=FI8;<=@E@K@FEF=C@K?@LDPFLD8P9<KFC;K?8K@K@JK?8K<C<D<EKN?FJ<8KFD@:N<@>?K@JM<IPE<8ICPLK@=K?<8LK?FI?8J8 DFI<CF>@:8CD@E;?<N@CCK<CCPFLK?8K@=PFLJ<8I:?8DFE>D@E<I8CJK?8K8I<M@KI<FLJKI8EJCL:<EK>I<PFI

 N?@K<M<IP?8I;9I@KKC<8E;@EJFCL9C<=FIFE<N?@:?

@DG8IKJ8:I@DJFEK@E><KF8ELECLD@EFLJ=C8D<K?@JD@E<I8C9<@E>KI@KLI8K<;N@K?C@D<FIN@K?<I@K<I8KJe98E<8E;K?<E=LJ<;:8E9<G8IKCP;@JJFCM<;@EDLI@8K@:8:@;8E;@=K?@JJFCLK@FE9<<M8GFI8K<;8E;K?<I<J@;L<9<<OKI8:K<;N@K?JLCG?LI@:8:@;8E;;LCPGLI@=@<;@K:8E9<:FEM<IK<;9PFI;@E8IPD<K?F;J@EKF8:?CFI@;<N?@:?9<@E>F9K8@E<;@EK?<JFC@;JK8K<=LJ<;8E;<C<:KIFCPQ<;

 N@K??8C=8;FQ<EGFN<I=LC:<CCJN@CCP@<C;8>CF9LC<F=8 G@EB@J?J@CM<IPD<K8CK?8KN@CC=CF8KFE>8JFC<E<8E;K?<

D8K<I@8CF=K?8K@J8JG<:@D<EF=C@K?@LDa g?8IC<J.8E;<IJ+<@I:<

/?8KcJ 8E <O8DGC< F= `CF>@:8C D@E;a 8J ;<J:I@9<; 9P 8 ><EL@E</I8;@K@FE8C-8K@FE8C@JKI8K?<IK?8E8#FCCPNFF;J:I@GKNI@K<I

LK EFK< +<@I:< @JEcK actually J?FN@E> PFL 8 G@<:< F= C@K?@LD#< ;@;EcK ?8M< G@<:<J F= C@K?@LD JK8GC<; KF ?@J 9FFB -8K?<I ?<cJ

 >@M@E> PFL 8 KI<8JLI< D8Gg8E @EK<EJ@FE8CCP ;<=@E<; GIF:<;LI<

 N?@:? N?<E <O<:LK<; N@CC C<8; PFL KF 8E <OK<EJ@FE8C <O8DGC< F= C@K?@LD /?@J @J EFK K?< J8D< 8J ALJK KFJJ@E> PFL 8 ?LEB F= C@K?@LD9LK @KcJ EFK K?< J8D< 8J J8P@E> `8KFD@: N<@>?K a <@K?<I h/?FL>? @= 

 PFL ?8; sufficiently sharp <P<J J8P@E> ` GIFKFEJa D@>?K C<K PFL G@:BFLKC@K?@LD8K8>C8E:<_i

.F K?8K @J @EK<EJ@FE8C 8E; <OK<EJ@FE8C definition., N?@:? @J 8 N8P F= K<CC@E> JFD<FE< <CJ< N?8K PFL D<8E 9P 8 :FE:<GK 2?<E $K8CB<; 89FLK `;<=@E@K@FEJa 89FM< $ K8CB<; 89FLK 8 N8P F= communi-cating :FE:<GKJg telling someone else N?8K PFL D<8E 9P `I<;a `K@><Ia

0,&+!'&+&! &,&+!'&+

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 202/309

`?LD8Ea FI `C@K?@LDa )FN C<KcJ K8CB 89FLK K?< 8:KL8C :FE:<GKJK?<DJ<CM<J

/?< 8:KL8C @EK<EJ@FE F= DP `K@><Ia :FE:<GK NFLC; 9< K?< E<LI8C

G8KK<IE h@E DP K<DGFI8C :FIK<Oi K?8K @EJG<:KJ 8E @E:FD@E> J@>E8C=IFDK?<M@JL8C:FIK<OKF;<K<ID@E<N?<K?<IFIEFK@K@J8K@><I

/?< 8:KL8C <OK<EJ@FE F= DP `K@><Ia :FE:<GK @J <M<IPK?@E> $ :8CC 8 K@><I

$EK<EJ@FE8C ;<=@E@K@FEJ ;FEcK :8GKLI< <EK@I< @EK<EJ@FEJ <OK<EeJ@FE8C ;<=@E@K@FEJ ;FEcK :8GKLI< <EK@I< <OK<EJ@FEJ $= $ GF@EK KF ALJKFE< K@><I 8E; J8P K?< NFI; `K@><Ia K?< :FDDLE@:8K@FE D8P =8@C @= K?<P K?@EB $ D<8E `;8E><IFLJ 8E@D8Ca FI `D8C< K@><Ia FI `P<CCFN K?@E>a .@D@C8ICP @= $ J8P `;8E><IFLJ P<CCFNe9C8:B JKI@G<; 8E@D8Ca

 N@K?FLK GF@EK@E> KF 8EPK?@E> K?< C@JK<E<I D8P M@JL8C@Q< >@8EK ?FIeE<KJ

4FL :8EcK :8GKLI< @E NFI;J 8CC K?< ;<K8@CJ F= K?< :F>E@K@M< :FEe:<GKg8J @K <O@JKJ @E PFLI D@E;gK?8K C<KJ PFL I<:F>E@Q< K?@E>J 8JK@><IJ FI EFEK@><IJ $KcJ KFF C8I>< E; PFL :8EcK GF@EK KF 8CC K?<K@><IJPFLcM<<M<IJ<<EC<K8CFE<<M<IPK?@E>PFL would :8CC8K@><I

/?< JKIFE><JK ;<=@E@K@FEJ LJ< 8 :IFJJ=@I< F= @EK<EJ@FE8C 8E; <OeK<EJ@FE8C :FDDLE@:8K@FE KF E8@C ;FNE 8 :FE:<GK M<E JF PFL FECP 

:FDDLE@:8K< maps to :FE:<GKJ FI @EJKIL:K@FEJ =FI 9L@C;@E> :FEe:<GKJgPFL ;FEcK :FDDLE@:8K< K?< actual :8K<>FI@<J 8J K?<P <O@JK @E

 PFLID@E;FI@EK?<NFIC;

h4<J N@K? <EFL>? :I<8K@M@KP PFL :8E :FEJKIL:K <O:<GK@FEJ KFK?@J ILC< C@B< `.<EK<E:<J C@<Q<I 4L;BFNJBP ?8J GL9C@J?<; :FEK8@Ee@E> K?< K<ID b?LI8>8CFE@c 8J F= !<9 a $cM< ALJK J?FNE PFL K?@J:FE:<GKcJ <EK@I< <OK<EJ@FE LK <O:<GK @E D8K?<D8K@:J ;<=@E@K@FEJ8I<LJL8CCPKI<8JLI<D8GJEFKKI<8JLI<i

.F K?8KcJ 8EFK?<I I<8JFE PFL :8EcK `;<=@E< 8 NFI; 8EP N8P PFLC@B<a 4FL :8EcK ;@I<:KCP GIF>I8D :FE:<GKJ @EKF JFD<FE< <CJ<cJ9I8@E

M<E N@K?@E K?< I@JKFK<C@8E G8I8;@>D N?<I< N< GI<K<E; K?8KK?< ;<=@E@K@FEJ 8I< K?< 8:KL8C :FE:<GKJ PFL ;FEcK ?8M< simultaneous=I<<;FD F= @EK<EJ@FE 8E; <OK<EJ@FE .LGGFJ< $ ;<=@E< (8IJ 8J `?L>< I<; IF:BP JG?<I< 8IFLE; 8 K<EK? F= 8IK?cJ D8JJ 8E; n=LIK?<I 8N8P =IFD K?< .LEa $KcJ K?<E 8 J<G8I8K< D8KK<I KF J?FN K?8K K?@J @EK<EJ@FE8C ;<=@E@K@FE D8K:?<J JFD< G8IK@:LC8I <OK<EJ@FEe

8C K?@E> @E DP <OG<I@<E:< FI @E;<<; K?8K @K D8K:?<J 8EP I<8C K?@E> 

-%&6+-!,'/'*+

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 203/309

 N?8KJF<M<I $= @EJK<8; $ J8P `/?8KcJ (8IJa 8E; GF@EK KF 8 I<; C@>?K@E K?< E@>?K JBP @K 9<:FD<J 8 J<G8I8K< D8KK<I KF J?FN K?8K K?@J <OeK<EJ@FE8C C@>?K D8K:?<J 8EP G8IK@:LC8I @EK<EJ@FE8C ;<=@E@K@FE $ D8P 

GIFGFJ<gFI 8EP @EK<EJ@FE8C 9<C@<=J $ D8P ?8M<gJL:? 8J `(8IJ @JK?<"F;F=28Ia

LK DFJK F= K?< 9I8@EcJ NFIB F= 8GGCP@E> @EK<EJ@FEJ ?8GG<EJJL9e;<C@9<I8K<CP 2< 8I<EcK :FEJ:@FLJCP 8N8I< K?8K FLI @;<EK@=@:8eK@FE F= 8 I<; C@>?K 8J `(8IJa @J 8 J<G8I8K< D8KK<I =IFD FLI M<I98C;<=@E@K@FE `(8IJ @J K?< "F; F= 28Ia )F D8KK<I N?8K B@E; F= @EeK<EJ@FE8C ;<=@E@K@FE $ D8B< LG KF ;<J:I@9< (8IJ DP D@E; 9<C@<M<JK?8K `(8IJa I<=<IJ KF K?@J K?@E>P  8E; K?8K @K @J K?< =FLIK? GC8E<K @EK?<.FC8I.PJK<D

2?<E PFL K8B< @EKF 8::FLEK K?< N8P K?< ?LD8E D@E; 8:KL8CCPGI8>D8K@:8CCP NFIBJ K?< EFK@FE `$ :8E ;<=@E< 8 NFI; 8EP N8P $ C@B<aJFFE 9<:FD<J `$ :8E 9<C@<M< 8EPK?@E> $ N8EK 89FLK 8 =@O<; J<K F= F9A<:KJa FI `$ :8E DFM< 8EP F9A<:K $ N8EK @E FI FLK F= 8 =@O<; D<De9<IJ?@G K<JKa %LJK 8J PFL :8EcK LJL8CCP :FEM<P 8 :FE:<GKcJ N?FC<@EK<EJ@FE @E NFI;J 9<:8LJ< @KcJ 8 9@> :FDGC@:8K<; E<LI8C D<D9<IeJ?@G K<JK PFL :8EcK control K?< :FE:<GKcJ <EK@I< @EK<EJ@FE 9<:8LJ< @KcJ8GGC@<; JL9e;<C@9<I8K<CP /?@J @J N?P 8I>L@E> K?8K 345 @J KIL< `9P 

;<=@E@K@FEa @J JF GFGLC8I $= ;<=@E@K@FE :?8E><J 9<?8M<; C@B< K?< <DeG@I@:8C ELCCFGJ K?<PcI< JLGGFJ<; KF 9< EF FE< NFLC; 9FK?<I 8I>L@E> K?<D LK 89LJ< ;<=@E@K@FEJ ALJK 8 C@KKC< 8E; K?<P KLIE @EKF D8>@:

 N8E;Jg@E8I>LD<EKJF=:FLIJ<EFK@EI<8C@KP

http://en.wikipedia.org/wiki/Image:Mars_Hubble.jpg 

0,&+!'&+&! &,&+!'&+

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 204/309

5. Similarity Clusters

Followup to OK<EJ@FEJ8E;$EK<EJ@FEJ

*E:< LGFE 8 K@D< K?< G?@CFJFG?<IJ F= +C8KFcJ :8;<DP :C8@D<;K?8K K?< 9<JK ;<=@E@K@FE F= ?LD8E N8J 8 `=<8K?<IC<JJ 9@G<;a @Fe

 ><E<J F= .@EFG< 8CJF :8CC<; @F><E<J K?< PE@: @J J8@; KF ?8M<GIFDGKCP <O?@9@K<; 8 GCL:B<; :?@:B<E 8E; ;<:C8I<; `#<I< @J +C8KFcJD8Ea /?< +C8KFE@JKJ GIFDGKCP :?8E><; K?<@I ;<=@E@K@FE KF `8 =<8K?<IC<JJ9@G<;N@K?9IF8;E8@CJa

)F ;@:K@FE8IP EF <E:P:CFG<;@8 ?8J <M<I C@JK<; 8CC K?< K?@E>JK?8K ?LD8EJ ?8M< @E :FDDFE 2< ?8M< I<; 9CFF; =@M< =@E><IJ FE

<8:? F= KNF ?8E;J 9FEP JBLCCJ G8@IJ F= :?IFDFJFD<Jg9LK K?<J8D< D@>?K 9< J8@; F= FK?<I 8E@D8C JG<:@<J 2< D8B< :FDGC<O KFFCJ KF D8B< :FDGC<O KFFCJ N< LJ< JPEK8:K@:8C :FD9@E8KFI@8C C8Ee

 >L8>< N< ?8IE<JJ :I@K@:8C =@JJ@FE I<8:K@FEJ 8J 8 JFLI:< F= <E<I>PK?<J< K?@E>J D8P J<IM< FLK KF J@E>C< FLK FECP ?LD8EJ 9LK EFK 8CC ?LeD8EJgD8EP F= LJ ?8M< E<M<I 9L@CK 8 =@JJ@FE I<8:KFI 2@K? K?< I@>?KJ<K F= E<:<JJ8IPe8E;eJL==@:@<EK ><E< J<HL<E:<J PFL :FLC; J@E>C< FLK8CC ?LD8EJ 8E; FECP ?LD8EJg8K C<8JK =FI EFNg9LK @K NFLC; JK@CC9<=8I=IFD all K?8K?LD8EJ?8M<@E:FDDFE

LK JF CFE> 8J PFL ;FEcK ?8GG<E KF 9< E<8I 8 GCL:B<; :?@:B<E J8P@E> `'FFB =FI =<8K?<IC<JJ 9@G<;Ja D8P J<IM< KF G@:B FLK 8 =<N ;FQ<E F= K?< G8IK@:LC8I K?@E>J K?8K 8I< ?LD8EJ 8J FGGFJ<; KF ?FLJ<J M8J<JJ8E;N@:?<J:8KJ:FCFIJFID8K?<D8K@:8CK?<FI<DJ

*E:< K?< ;<=@E@K@FE `=<8K?<IC<JJ 9@G<;a ?8J 9<<E 9FLE; KF JFD< particular =<8K?<IC<JJ 9@G<;J PFL :8E CFFB FM<I K?< >IFLG 8E; 9<>@E?8IM<JK@E> JFD< F= K?< other :?8I8:K<I@JK@:Jg9<PFE; D<I< =<8K?<Ie=I<< KNFC<>>@E<JJgK?8K K?< `=<8K?<IC<JJ 9@G<;Ja J<<D KF J?8I< @E:FDDFE /?< G8IK@:LC8I =<8K?<IC<JJ 9@G<;J K?8K PFL J<< J<<D KF 8CeJF LJ< C8E>L8>< 9L@C; :FDGC<O KFFCJ JG<8B :FD9@E8KFI@8C C8E>L8><

 N@K? JPEK8O 9C<<; I<; 9CFF; @= GFB<; ;@< N?<E K?<P ;I@EB ?<DCF:B

/?LJ K?< :8K<>FIP `?LD8Ea >IFNJ I@:?<I 8E; 8;;J DFI< 8E;DFI< :?8I8:K<I@JK@:J 8E; N?<E @F><E<J =@E8CCP GI<J<EKJ ?@JGCL:B<; :?@:B<E N< 8I< EFK =FFC<; /?@J GCL:B<; :?@:B<E @J F9M@eFLJCPEFKJ@D@C8IKFK?<FK?<I`=<8K?<IC<JJ9@G<;Ja

http://lesswrong.com/lw/nj/similarity_clusters/  +8><bOK<EJ@FEJ8E;$EK<EJ@FEJc

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 205/309

h$= I@JKFK<C@8E CF>@: N<I< 8 >FF; DF;<C F= ?LD8E GJP:?FCF>PK?< +C8KFE@JKJ NFLC; ?8M< CFFB<; 8K K?< GCL:B<; :?@:B<E 8E; J8@;`4<JK?8KcJ8?LD8EN?8KcJPFLIGF@EKai

$= K?< first =<8K?<IC<JJ 9@G<; PFL J<< @J 8 GCL:B<; :?@:B<E K?<E PFL D8P <E; LG K?@EB@E> K?8K K?< M<I98C C89<C `?LD8Ea ;<EFK<J8 GCL:B<; :?@:B<E JF $ :8E DF;@=P DP KI<8JLI< D8G KF GF@EK KF`=<8K?<IC<JJ 9@G<;J N@K? 9IF8; E8@CJa 8E; @= $ 8D N@J< >F FE KF J8P`.<< @F><E<J FM<I K?<I< /?8KcJ 8 ?LD8E 8E; $cD 8 ?LD8E 8E;

 PFLcI< 8 ?LD8E 8E; K?8K :?@DG8EQ<< @J EFK 8 ?LD8E K?FL>? =8@ICP :CFJ<a

/?< @E@K@8C :CL< FECP ?8J KF C<8; K?< LJ<I KF K?< J@D@C8I@KP :CLJeK<IgK?< >IFLG F= K?@E>J K?8K ?8M< D8EP :?8I8:K<I@JK@:J @E :FDe

DFE =K<I K?8K K?< @E@K@8C :CL< ?8J J<IM<; @KJ GLIGFJ< 8E; $ :8E >FFE KF :FEM<P K?< E<N @E=FID8K@FE `?LD8EJ 8I< :LII<EKCP DFIK8CaFIN?8K<M<I<CJ<$N8EKKFJ8P89FLKLJ=<8K?<IC<JJ9@G<;J

  ;@:K@FE8IP @J 9<JK K?FL>?K F= EFK 8J 8 9FFB F= I@JKFK<C@8E:C8JJ ;<=@E@K@FEJ 9LK 8 9FFB F= ?@EKJ =FI D8K:?@E> M<I98C C89<CJ KFJ@D@C8I@KP :CLJK<IJ FI D8K:?@E> C89<CJ KF GIFG<IK@<J K?8K 8I< LJ<=LC@E;@JK@E>L@J?@E>J@D@C8I@KP:CLJK<IJ

+!%!$*!,1$-+,*+

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 206/309

6. Typicality and Asymmetrical Similarity 

Followup to .@D@C8I@KPCLJK<IJ

@I;J =CP 2<CC <O:<GK FJKI@:?<J ;FEcK LK N?@:? @J 8 DFI< KPGe@:8C9@I;g8IF9@EFI8EFJKI@:?2?@:? @J 8 DFI< KPG@:8C :?8@I ;<JB :?8@I 8 IF:B@E> :?8@I FI 8 9<8E98>:?8@I

(FJK G<FGC< NFLC; J8P K?8K 8 IF9@E @J 8 DFI< KPG@:8C 9@I; 8E; 8 ;<JB :?8@I @J 8 DFI< KPG@:8C :?8@I /?< :F>E@K@M< GJP:?FCF>@JKJ N?FJKL;P K?@J JFIK F= K?@E> <OG<I@D<EK8CCP ;F JF LE;<I K?< ?<8;@E> F= `KPG@:8C@KP <==<:KJa FI `GIFKFKPG< <==<:KJa h-FJ:? 8E; 'CFP; i

!FI <O8DGC< @= PFL 8JB JL9A<:KJ KF GI<JJ 8 9LKKFE KF @E;@:8K< `KIL<aFI `=8CJ<a @E I<JGFEJ< KF JK8K<D<EKJ C@B< ` IF9@E @J 8 9@I;a FI `G<E>L@E @J 8 9@I;a I<8:K@FE K@D<J 8I< =8JK<I =FI DFI< :<EKI8C <Oe8DGC<J h$cD JK@CC LEG8:B@E> DP 9FFBJ 9LK $cD I<8JFE89CP JLI< DP JFLI:< FE K?@J @J '8BF== i /PG@:8C@KP D<8JLI<J :FII<C8K< N<CCLJ@E> ;@==<I<EK @EM<JK@>8K@M< D<K?F;JgI<8:K@FE K@D<J 8I< FE< <Oe8DGC< PFL :8E 8CJF 8JB G<FGC< KF ;@I<:KCP I8K< FE 8 J:8C< F= KF ?FN N<CC 8E <O8DGC< hC@B< 8 JG<:@=@: IF9@Ei =@KJ 8 :8K<>FIP hC@B<`9@I;ai

.F N< ?8M< 8 D<EK8C D<8JLI< F= KPG@:8C@KPgN?@:? D@>?K G<Ie?8GJ =LE:K@FE 8J 8 ?<LI@JK@:g9LK @J K?<I< 8 :FII<JGFE;@E> 9@8J N<:8ELJ<KFG@E@K;FNE

2<CC N?@:? F= K?<J< JK8K<D<EKJ JKI@B<J PFL 8J DFI< E8KLI8C` @J 8GGIFO@D8K<CP a FI ` @J 8GGIFO@D8K<CP a $= PFLcI<C@B< DFJK G<FGC< K?< =@IJK JK8K<D<EK J<<DJ KF D8B< DFI< J<EJ<h.8;F:B i !FI J@D@C8I I<8JFEJ G<FGC< 8JB<; KF I8K< ?FN J@D@C8I(<O@:F @J KF K?< 0E@K<; .K8K<J >8M< :FEJ@JK<EKCP ?@>?<I I8K@E>JK?8E G<FGC< 8JB<; KF I8K< ?FN J@D@C8I K?< 0E@K<; .K8K<J @J KF (<O@e:F h/M<IJBP8E;"8K@i

 E; @= K?8K JK@CC J<<DJ ?8IDC<JJ 8 JKL;P 9P -@GJ hi J?FN<;K?8K G<FGC< N<I< DFI< C@B<CP KF <OG<:K 8 ;@J<8J< NFLC; JGI<8; =IFDIF9@EJ KF ;L:BJ FE 8E @JC8E; K?8E =IFD ;L:BJ KF IF9@EJ )FN K?@J@J EFK 8 logical @DGFJJ@9@C@KP 9LK @E 8 GI8>D8K@: J<EJ< N?8K<M<I ;@=e=<I<E:< J<G8I8K<J 8 ;L:B =IFD 8 IF9@E 8E; NFLC; D8B< 8 ;@J<8J< C<JJC@B<CP KF JGI<8; =IFD 8 ;L:B KF 8 IF9@E DLJK 8CJF 9< 8 ;@==<I<E:<

http://lesswrong.com/lw/nk/typicality_and_asymmetrical_similarity/  +8><b.@D@C8I@KPCLJK<IJc

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 207/309

9<KN<<E 8 IF9@E 8E; 8 ;L:B 8E; NFLC; D8B< 8 ;@J<8J< C<JJ C@B<CP KFJGI<8;=IFD8IF9@EKF8;L:B

4<J PFL :8E :FD< LG N@K? I8K@FE8C@Q8K@FEJ C@B< `2<CC K?<I<

:FLC; 9< DFI< E<@>?9FI@E> JG<:@<J F= K?< IF9@EJ N?@:? NFLC; D8B<K?< ;@J<8J< DFI< C@B<CP KF JGI<8; @E@K@8CCP <K:a 9LK 9< :8I<=LC EFKKF KIP KFF ?8I; KF I8K@FE8C@Q< K?< GIF989@C@KP I8K@E>J F= JL9A<:KJ N?F;@;EcK <M<E I<8C@Q< K?<I< N8J 8 :FDG8I@JFE >F@E> FE E; ;FEcK=FI><K K?8K (<O@:F @J DFI< J@D@C8I KF K?< 0E@K<; .K8K<J K?8E K?<0E@K<; .K8K<J @J KF (<O@:F 8E; K?8K @J :CFJ<I KF K?8E @JKF J@DGC<I @EK<IGI<K8K@FE @J K?8K G<FGC< 8I< LJ@E> K?< h;<DFEeJKI8K<;i J@D@C8I@KP ?<LI@JK@: 8J 8 GIFOP =FI K?< GIF989@C@KP K?8K 8 ;@J<8J<JGI<8;J8E;K?@J?<LI@JK@:@Jh;<DFEJKI89CPi8JPDD<KI@:8C

&8EJ8J @J LELJL8CCP :CFJ< KF K?< :<EK<I F= K?< 0E@K<; .K8K<J8E; C8JB8 @J LELJL8CCP =8I =IFD K?< :<EK<I F= K?< 0E@K<; .K8K<JJF &8EJ8J @J GIF989CP :CFJ<I KF DFJK GC8:<J @E K?< 0. 8E; C8JB8 @J GIF989CP =8IK?<I $K ;F<J EFK =FCCFN ?FN<M<I K?8K &8EJ8J @J:CFJ<I KF C8JB8 K?8E @J C8JB8 KF &8EJ8J LK G<FGC< J<<D KF I<8eJFE hD<K8G?FI@:8CCP JG<8B@E>i 8J @= :CFJ<E<JJ @J 8E @E?<I<EK GIFG<IKP F= &8EJ8J 8E; ;@JK8E:< @J 8E @E?<I<EK GIFG<IKP F= C8JB8 JF K?8K&8EJ8J @J JK@CC :CFJ< <M<E KF C8JB8 8E; C8JB8 @J JK@CC ;@JK8EK <M<E=IFD&8EJ8J

.F FE:< 8>8@E N< J<< K?8K I@JKFKC<cJ EFK@FE F= :8K<e >FI@<JgCF>@:8C :C8JJ<J N@K? D<D9<IJ?@G ;<K<ID@E<; 9P 8 :FCC<:K@FEF= GIFG<IK@<J K?8K 8I< @E;@M@;L8CCP JKI@:KCP E<:<JJ8IP 8E; KF><K?<IJKI@:KCP JL==@:@<EKg@J EFK 8 >FF; DF;<C F= ?LD8E :F>E@K@M< GJPe:?FCF>P h.:@<E:<cJ M@<N ?8J :?8E><; JFD<N?8K FM<I K?< C8JK

 P<8IJ 2?F NFLC;cM< K?FL>?Ki 2< ;FEcK <M<E I<8JFE 8J @= J<KD<D9<IJ?@G @J 8 KIL<eFIe=8CJ< GIFG<IKP .K8K<D<EKJ F= J<K D<D9<IeJ?@G :8E 9< DFI< FI C<JJ KIL< h)FK< /?@J @J not K?< J8D< K?@E> 8J

9<@E>DFI<FIC<JJGIF989C<i*E< DFI< I<8JFE EFK KF GI<K<E; K?8K PFL FI 8EPFE< <CJ< @J re-

 ally >F@E>KFKI<8KNFI;J8JI@JKFK<C@8ECF>@:8C:C8JJ<J

'8BF== "<FI>< hi Women, Fire and Dangerous Things: What Categories Tell Us About the Nature of Thought  0E@M<IJ@KP F= ?@:8>F+I<JJ?@:8>F

+8><b2FI;J8J#@;;<E$E=<I<E:<Jc

,1(!$!,1&+1%%,*!$+!%!$*!,1

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 208/309

-@GJ '8E:< % hi `$E;L:K@M< AL;>D<EKJ 89FLK E8KLI8C :8K<e >FI@<Ja Journal of Verbal Learning and Verbal Behavior. 14e

-FJ:? C<8EFI 8E; 'CFP; <;J hi Cognition and Cate-

 gorization. #@CCJ;8C<)%'8NI<E:<IC98LDJJF:@8K<J.8;F:B %<IIFC; hi `/ILK? 8E; GGIFO@D8K@FEJa $E Papers

 from the Third Annual Meeting of the Berkeley Linguistics Society, GGe <IB<C<P<IB<C<P'@E>L@JK@:J.F:@<KP

/M<IJBP DFJ 8E; $K8D8I "8K@ hi `.KL;@<J F= .@D@C8I@KPa$E-FJ:?8E;'CFP;hi

-%&6+-!,'/'*+

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 209/309

7. The Cluster Structure of Thingspace

Followup to /PG@:8C@KP8E;JPDD<KI@:8C.@D@C8I@KP 

/?< EFK@FE F= 8 `:FE=@>LI8K@FE JG8:<a @J 8 N8P F= KI8EJC8K@E> F9eA<:K descriptions @EKF F9A<:K positions. $K D8P J<<D C@B< 9CL< @J `:CFJ<IaKF 9CL<e>I<<E K?8E KF I<; 9LK ?FN DL:? :CFJ<I $KcJ ?8I; KF 8EeJN<I K?8K HL<JK@FE 9P ALJK JK8I@E> 8K K?< :FCFIJ LK @K ?<CGJ KF BEFN K?8K K?< hGIFGFIK@FE8Ci :FCFI :FFI;@E8K<J @E -" 8I< 8E; $KNFLC;9<<M<E:C<8I<I@=GCFKK<;FE8>I8G?

$E K?< J8D< N8P PFL :8E J<< 8 IF9@E 8J 8 IF9@Eg9IFNE K8@C I<;9I<8JK JK8E;8I; IF9@E J?8G< D8O@DLD =CP@E> JG<<; N?<E LEC8;<E

@KJ JG<:@<JeKPG@:8C ) 8E; @E;@M@;L8C 8CC<C<J *I PFL :FLC; J<< 8 IF9@E 8J 8 J@E>C< GF@EK @E 8 :FE=@>LI8K@FE JG8:< N?FJ< ;@D<EJ@FEJ;<J:I@9<;<M<IPK?@E>N<BE<NFI:FLC;BEFN89FLKK?<IF9@E

  IF9@E @J 9@>><I K?8E 8 M@ILJ 8E; JD8CC<I K?8E 8E 8@I:I8=K:8II@<IgK?8K D@>?K 9< K?< `MFCLD<a ;@D<EJ@FE '@B<N@J< 8 IF9@E

 N<@>?J DFI< K?8E 8 ?P;IF><E 8KFD 8E; C<JJ K?8E 8 >8C8OP K?8KD@>?K 9< K?< `D8JJa ;@D<EJ@FE @==<I<EK IF9@EJ N@CC ?8M< JKIFE> :FII<C8K@FEJ 9<KN<<E `MFCLD<a 8E; `D8JJa JF K?< IF9@EeGF@EKJ N@CC9< C@E<; LG @E 8 =8@ICP C@E<8I JKI@E> @E K?FJ< KNF ;@D<EJ@FEJg9LK

K?< :FII<C8K@FE NFEcK 9< <O8:K JF N< ;F E<<; KNF J<G8I8K< ;@D<EeJ@FEJ

/?@J @J K?< 9<E<=@K F= M@<N@E> IF9@EJ 8J GF@EKJ @E JG8:< 4FL:FLC;EcK J<< K?< C@E<8I C@E<LG 8J <8J@CP @= PFL N<I< ALJK @D8>@E@E> K?<IF9@EJ8J:LK<C@KKC<N@E>e=C8GG@E>:I<8KLI<J

  IF9@EcJ ) @J 8 ?@>?CP DLCK@;@D<EJ@FE8C M8I@89C< 9LK PFL:8E JK@CC K?@EB F= @K 8J G8IK F= 8 IF9@EcJ CF:8K@FE @EK?@E>JG8:<gD@CC@FEJ F= HL8K<IE8IP :FFI;@E8K<J FE< :FFI;@E8K< =FI

<8:? ) 98J<gFI D8P9< 8 DFI< JFG?@JK@:8K<; M@<N K?8K /?<J?8G< F= K?< IF9@E 8E; @KJ :FCFI hJLI=8:< I<=C<:K8E:<i PFL :8E C@B<e N@J< K?@EB F= 8J G8IK F= K?< IF9@EcJ GFJ@K@FE @E K?@E>JG8:< <M<EK?FL>?K?<P8I<EcK single ;@D<EJ@FEJ

 %LJK C@B< K?< :FFI;@E8K< GF@EK :FEK8@EJ K?< J8D< @E=FIeD8K@FE 8J K?< 8:KL8C #/(' :FCFI 9CL< N< J?FLC;EcK 8:KL8CCP CFJ<@E=FID8K@FE N?<E N< J<< IF9@EJ 8J GF@EKJ @E JG8:< 2< 9<C@<M< K?<J8D< JK8K<D<EK 89FLK K?< IF9@EcJ D8JJ N?<K?<I N< M@JL8C@Q< 8 IF9@E

http://lesswrong.com/lw/nl/the_cluster_structure_of_thingspace/  +8><b/PG@:8C@KP8E;JPDD<KI@:8C.@D@C8I@KPc

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 210/309

98C8E:@E> K?< J:8C<J FGGFJ@K< 8 eB@CF>I8D N<@>?K FI 8 IF9@EeGF@EKN@K?8D8JJe:FFI;@E8K<F=

2< :8E <M<E @D8>@E< 8 :FE=@>LI8K@FE JG8:< N@K? FE< FI DFI<

;@D<EJ@FEJ =FI <M<IP ;@JK@E:K :?8I8:K<I@JK@: F= 8E F9A<:K JF K?8K K?< position F= 8E F9A<:KcJ GF@EK @E K?@J JG8:< :FII<JGFE;J KF all K?< @Ee=FID8K@FE @E K?< I<8C F9A<:K @KJ<C= -8K?<I I<;LE;8EKCP I<GI<J<EK<;KFFg;@D<EJ@FEJ NFLC; @E:CL;< K?< D8JJ K?< MFCLD< 8E; K?< ;<EeJ@KP

$= PFL K?@EB K?8KcJ <OKI8M8>8EK HL8EKLD G?PJ@:@JKJ LJ< 8Einfinite-dimensional  :FE=@>LI8K@FE JG8:< 8E; 8 J@E>C< GF@EK @E K?8KJG8:< ;<J:I@9<J K?< CF:8K@FE F= <M<IP G8IK@:C< @E K?< LE@M<IJ< .F

 N<cI< 8:KL8CCP 9<@E> :FDG8I8K@M<CP :FEJ<IM8K@M< @E FLI M@JL8C@Q8K@FE

F=  thingspace g8 GF@EK @E K?@E>JG8:< ;<J:I@9<J ALJK FE< F9A<:K EFKK?<<EK@I<LE@M<IJ<

$= N<cI< EFK JLI< F= K?< IF9@EcJ <O8:K D8JJ 8E; MFCLD< K?<E N<:8E K?@EB F= 8 C@KKC< :CFL; @E K?@E>JG8:< 8 volume of uncertainty N@K?e@E N?@:? K?< IF9@E D@>?K 9< /?< ;<EJ@KP F= K?< :CFL; @J K?< ;<EJ@KP F= FLI 9<C@<= K?8K K?< IF9@E ?8J K?8K G8IK@:LC8I D8JJ 8E; MFCLD< $= 

 PFLcI< DFI< JLI< F= K?< IF9@EcJ ;<EJ@KP K?8E F= @KJ D8JJ 8E; MFCLD< PFLI GIF989@C@KPe:CFL; N@CC 9< ?@>?CP :FE:<EKI8K<; @E K?< ;<EJ@KP ;@e

D<EJ@FE 8E; :FE:<EKI8K<; 8IFLE; 8 JC8EK@E> C@E< @E K?< JL9JG8:< F= D8JJMFCLD< h$E;<<; K?< :CFL; ?<I< @J 8:KL8CCP 8 JLI=8:< 9<:8LJ<F=K?<I<C8K@FE1(i

`-8;@8C :8K<>FI@<Ja 8I< ?FN :F>E@K@M< GJP:?FCF>@JKJ ;<J:I@9< K?<EFEeI@JKFK<C@8E 9FLE;8I@<J F= NFI;J /?< :<EKI8C `DFK?<Ia :FEe:<@M<J ?<I :?@C; >@M<J 9@IK? KF @K 8E; JLGGFIKJ @K $J 8E <>> ;FEFI

 N?F E<M<I J<<J ?<I :?@C; 8 DFK?<I .?< @J K?< `><E<K@: DFK?<Ia2?8K 89FLK 8 NFD8E N?F @J @DGC8EK<; N@K? 8 =FI<@>E <D9IPF 8E;9<8IJ @K KF K<ID .?< @J 8 `JLIIF>8K< DFK?<Ia E; K?< NFD8E N?F

I8@J<J 8 :?@C; K?8K @JEcK ?<IJ ><E<K@:8CCP 2?P J?<cJ 8E `8;FGK@M<DFK?<Ia /?< I@JKFK<C@8E JPCCF>@JD NFLC; ILE `#LD8EJ ?8M< K<E=@E><IJ !I<; ?8J E@E< =@E><IJ K?<I<=FI< !I<; @J EFK 8 ?LD8Ea 9LKK?< N8P N< 8:KL8CCP K?@EB @J `#LD8EJ ?8M< K<E =@E><IJ !I<; @J 8 ?LeD8EK?<I<=FI<!I<;@J8bE@E<e=@E><I<;?LD8Eca

2< :8E K?@EB 89FLK K?< I8;@8CeE<JJ F= :8K<>FI@<J @E @EK<EJ@FE8CK<IDJ 8J ;<J:I@9<; 89FM<gGIFG<IK@<J K?8K 8I< LJL8CCP GI<J<EK 9LKFGK@FE8CCP 89J<EK $= N< K?FL>?K 89FLK K?< @EK<EJ@FE F= K?< NFI;`DFK?<Ia @K D@>?K 9< C@B< 8 ;@JKI@9LK<; >CFN @E K?@E>JG8:< 8 >CFN 

-%&6+-!,'/'*+

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 211/309

 N?FJ< @EK<EJ@KP D8K:?<J K?< ;<>I<< KF N?@:? K?8K MFCLD< F= K?@E>JG8:< D8K:?<J K?< :8K<>FIP `DFK?<Ia /?< >CFN @J :FE:<EKI8Ke<; @E K?< :<EK<I F= ><E<K@:J 8E; 9@IK? 8E; :?@C;eI8@J@E> K?< MFCLD<

F=<>>;FEFIJNFLC;8CJF>CFN9LKC<JJ9I@>?KCP*I N< :8E K?@EB 89FLK K?< I8;@8CeE<JJ F= :8K<>FI@<J <OK<EJ@FE8Ce

CP .LGGFJ< N< D8GG<; 8CC K?< 9@I;J @E K?< NFIC; @EKF K?@E>JG8:<LJ@E> 8 ;@JK8E:< D<KI@: K?8K :FII<JGFE;J 8J N<CC 8J GFJJ@9C< KF G<Ie:<@M<; J@D@C8I@KP @E ?LD8EJ IF9@E @J DFI< J@D@C8I KF 8EFK?<IIF9@E K?8E <@K?<I @J J@D@C8I KF 8 G@><FE 9LK IF9@EJ 8E; G@><FEJ 8I<8CCDFI<J@D@C8IKF<8:?FK?<IK?8E<@K?<I@JKF8G<E>L@E<K:<K<I8

/?<E K?< :<EK<I F= 8CC 9@I;E<JJ NFLC; 9< ;<EJ<CP GFGLC8K<; 9P D8EP E<@>?9FI@E> K@>?K :CLJK<IJ IF9@EJ 8E; JG8IIFNJ 8E; :8E8I@<J

8E; G@><FEJ 8E; D8EP FK?<I JG<:@<J 8>C<J 8E; =8C:FEJ 8E; FK?e<I C8I>< GI<;8KFIP 9@I;J NFLC; F::LGP 8 E<8I9P :CLJK<I +<E>L@EJ

 NFLC; 9< @E 8 DFI< ;@JK8EK :CLJK<I 8E; C@B<N@J< :?@:B<EJ 8E; FJeKI@:?<J

/?< I<JLCK D@>?K CFFB @E;<<; JFD<K?@E> C@B< 8E 8JKIFEFD@:8C:CLJK<ID8EP>8C8O@<JFI9@K@E>K?<:<EK<I8E;8=<NFLKC@<IJ

*I N< :FLC; K?@EB J@DLCK8E<FLJCP 89FLK 9FK? K?< @EK<EJ@FE F= K?< :F>E@K@M< :8K<>FIP `9@I;a 8E; @KJ <OK<EJ@FE @E I<8CeNFIC; 9@I;J

/?< :<EKI8C :CLJK<IJ F= IF9@EJ 8E; JG8IIFNJ >CFN@E> 9I@>?KCP N@K??@>?CP KPG@:8C 9@I;E<JJ J8K<CC@K< :CLJK<IJ F= FJKI@:?<J 8E; G<E>L@EJ >CFN@E> DFI< ;@DCP N@K? 8KPG@:8C 9@I;E<JJ 8E; 9I8?8D '@E:FCE 8 =<ND<>8G8IJ<:J8N8P8E;>CFN@E>EFK8K8CC

$ GI<=<I K?8K C8JK M@JL8C@Q8K@FEgK?< >CFN@E> GF@EKJg9<:8LJ< 8J$ J<< @K K?< JKIL:KLI< F= K?< :F>E@K@M< @EK<EJ@FE =FCCFN<; =IFDK?< <OK<EJ@FE8C :CLJK<I JKIL:KLI< !@IJK :8D< K?< JKIL:KLI<e@EeK?<e

 NFIC; K?< <DG@I@:8C ;@JKI@9LK@FE F= 9@I;J FM<I K?@E>JG8:< K?<E 9P F9J<IM@E> @K N< =FID<; 8 :8K<>FIP N?FJ< @EK<EJ@FE8C >CFN IFL>?CP 

FM<IC8PJK?@JJKIL:KLI</?@J >@M<J LJ P<K 8EFK?<I M@<N F= N?P NFI;J 8I< EFK I@JKFK<C@8E

:C8JJ<J K?< <DG@I@:8C :CLJK<I<; JKIL:KLI< F= K?< I<8C LE@M<IJ< @J EFKJF :IPJK8CC@E< E8KLI8C :CLJK<I 8 >IFLG F= K?@E>J ?@>?CP J@D@C8IKF <8:? FK?<I D8P ?8M< no J<K F= E<:<JJ8IP 8E; JL==@:@<EK GIFG<IeK@<Jg  no J<K F= :?8I8:K<I@JK@:J K?8K 8CC >IFLG D<D9<IJ ?8M< 8E; EFEFEeD<D9<IJ?8M<

LK <M<E @= 8 :8K<>FIP @J @II<:FM<I89CP 9CLIIP 8E; 9LDGP K?<I<cJ

EF E<<; KF G8E@: $ NFLC; EFK F9A<:K @= JFD<FE< J8@; K?8K 9@I;J

,$-+,* +,*-,-*',!&+(

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 212/309

8I< `=<8K?<I<; =CP@E> K?@E>Ja But penguins don’t fly!  gN<CC =@E< /?<LJL8C ILC< ?8J 8E <O:<GK@FE @KcJ EFK K?< <E; F= K?< NFIC; <=@E@eK@FEJ :8EcK 9< <OG<:K<; KF <O8:KCP D8K:? K?< <DG@I@:8C JKIL:KLI< F= 

K?@E>JG8:< @E 8EP <M<EK 9<:8LJ< K?< D8G @J JD8CC<I 8E; DL:? C<JJ:FDGC@:8K<; K?8E K?< K<II@KFIP /?< GF@EK F= K?< ;<=@E@K@FE `=<8K?e<I<; =CP@E> K?@E>Ja @J KF C<8; K?< C@JK<E<I KF K?< 9@I; :CLJK<I EFK KF

 >@M< 8 KFK8C ;<J:I@GK@FE F= <M<IP <O@JK@E> 9@I; ;FNE KF K?< DFC<:LC8IC<M<C

2?<E PFL ;I8N 8 9FLE;8IP 8IFLE; 8 >IFLG F= <OK<EJ@FE8CGF@EKJ empirically :CLJK<I<; @E K?@E>JG8:< PFL D8P =@E; 8K C<8JK FE<<O:<GK@FEKF<M<IPJ@DGC<@EK<EJ@FE8CILC<PFL:8E@EM<EK

LK @= 8 ;<=@E@K@FE NFIBJ N<CC <EFL>? @E GI8:K@:< KF GF@EK FLK

K?< @EK<E;<; <DG@I@:8C :CLJK<I F9A<:K@E> KF @K D8P ALJKCP 9< :8CC<;`E@KG@:B@E>a

-%&6+-!,'/'*+

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 213/309

8. Disguised Queries

Followup to /?<CLJK<I.KIL:KLI<F=/?@E>JG8:<

$D8>@E< K?8K PFL ?8M< 8 G<:LC@8I AF9 @E 8 G<:LC@8I =8:KFIP 4FLIK8JB @J KF K8B< F9A<:KJ =IFD 8 DPJK<I@FLJ :FEM<PFI 9<CK 8E; JFIKK?< F9A<:KJ @EKF KNF 9@EJ 2?<E PFL =@IJK 8II@M< .LJ8E K?< .<[email protected]<I <OGC8@EJ KF PFL K?8K 9CL< <>>eJ?8G<; F9A<:KJ 8I< :8CC<; `9C<>e

 >Ja 8E; >F @E K?< `9C<>> 9@Ea N?@C< I<; :L9<J 8I< :8CC<; `IL9<Ja 8E; >F@EK?<`IL9<9@Ea

*E:< PFL JK8IK NFIB@E> PFL EFK@:< K?8K 9C<>>J 8E; IL9<J ;@==<I@E N8PJ 9<J@;<J :FCFI 8E; J?8G< C<>>J ?8M< =LI FE K?<@I JLI=8:<

 N?@C< IL9<J 8I< JDFFK? C<>>J =C<O JC@>?KCP KF K?< KFL:? IL9<J 8I<?8I; C<>>J8I<FG8HL<K?<IL9<cJJLI=8:<JC@>?KCPKI8EJCL:<EK

.FFE 8=K<I PFL 9<>@E NFIB@E> PFL <E:FLEK<I 8 9C<>> J?8;<;8E LELJL8CCP ;8IB 9CL<g@E =8:K FE :CFJ<I <O8D@E8K@FE K?< :FCFIGIFM<JKF9<GLIGC<?8C=N8P9<KN<<EI<;8E;9CL<

4<K N8@K 2?P 8I< PFL :8CC@E> K?@J F9A<:K 8 `9C<>>a `9C<>>a N8J FI@>@E8CCP ;<=@E<; 8J 9CL< 8E; <>>eJ?8G<;gK?< HL8C@=@:8K@FE F= 9CL<E<JJ 8GG<8IJ @E K?< M<IP E8D< `9C<>>a @E =8:K /?@J F9A<:K @JEFK 9CL< *E< F= K?< E<:<JJ8IP HL8C@=@:8K@FEJ @J D@JJ@E> PFL J?FLC;

:8CCK?@J8`GLIGC<<>>eJ?8G<;F9A<:KaEFK8`9C<>>aLK @K JF ?8GG<EJ K?8K @E 8;;@K@FE KF 9<@E> GLIGC< 8E; <>>e

J?8G<; K?< F9A<:K @J 8CJF =LII<; =C<O@9C< 8E; FG8HL< .F N?<E PFLJ8N K?< F9A<:K PFL K?FL>?K `*? 8 JKI8E><CP :FCFI<; 9C<>>a $K :<IeK8@ECP@JEcK8IL9<_I@>?K

.K@CC PFL 8I<EcK HL@K< JLI< N?8K KF ;F E<OK .F PFL :8CC FM<I .LeJ8EK?<.<[email protected]<I

`*?P<J@KcJ89C<>>a.LJ8EJ8PJ`PFL:8EGLK@K@EK?<9C<>>9@Ea4FLJK8IKKFKFJJK?<GLIGC<9C<>>@EKFK?<9C<>>9@E

9LKG8LJ<=FI8DFD<EK `.LJ8EaPFLJ8P`?FN;FPFL know K?@J@J89C<>>a

.LJ8ECFFBJ8KPFLF;;CP `$JEcK@KF9M@FLJ /?@JF9A<:KD8P9<GLIGC<9LK@KcJJK@CC<>>eJ?8G<;=LII<;=C<O@9C<8E;FG8HL<C@B<8CCK?<FK?<I9C<>>J 4FLcM<>FK

http://lesswrong.com/lw/nm/disguised_queries/  +8><b/?<CLJK<I.KIL:KLI<F=/?@E>JG8:<c

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 214/309

KF<OG<:K8=<N:FCFI;<=<:KJ *I@JK?@JFE<F=K?FJ<G?@CFJFG?@:8C:FELE;ILDJC@B<b#FN;FPFLBEFNK?<

 NFIC;N8JEcK:I<8K<;=@M<D@ELK<J8>F:FDGC<K<N@K?

=8CJ<D<DFI@<Jc $E8G?@CFJFG?@:8CJ<EJ<$cDEFK absolutely certain K?8KK?@J@J89C<>>9LK@KJ<<DJC@B<8  >FF;>L<JJa

`)F$D<8E_a 4FLG8LJ<J<8I:?@E>=FINFI;J `Why@JK?<I<89C<>>9@E8E;8IL9<9@E 2?8KcJK?< difference9<KN<<E9C<>>J8E;IL9<Ja

`C<>>J8I<9CL<8E;<>>eJ?8G<;IL9<J8I<I<;8E;:L9<eJ?8G<;a.LJ8EJ8PJG8K@<EKCP `4FL>FKK?<JK8E;8I;FI@<EK8K@FEC<:KLI<I@>?Ka

`2?P;F9C<>>J8E;IL9<J need KF9<JFIK<;a`I_9<:8LJ<FK?<IN@J<K?<Pc;9<8CCD@O<;LGaJ8PJ

.LJ8E `<:8LJ<EF9F;PN@CCG8PLJKFJ@K8IFLE;8CC;8P 8E; not JFIK9C<>>J8E;IL9<Ja

`2?FFI@>@E8CCP;<K<ID@E<;K?8KK?<=@IJK9CL<<>>eJ?8G<;F9A<:KN8J8b9C<>>c8E;?FN;@;K?<P;<K<ID@E<K?8Ka

.LJ8EJ?IL>J `$JLGGFJ<PFL:FLC;ALJK8J<8J@CP:8CCK?<I<;:L9<eJ?8G<;F9A<:KJb9C<>>Jc8E;K?<9CL<<>>e

J?8G<;F9A<:KJbIL9<Jc9LK@KJ<<DJ<8J@<IKFI<D<D9<IK?@JN8Pa

4FLK?@EB=FI8DFD<EK `.LGGFJ<8:FDGC<K<CP D@O<;eLGF9A<:K:8D<F==K?<:FEM<PFI '@B<8EFI8E><JG?<I<eJ?8G<;=LII<;KI8EJCL:<EKF9A<:KN@K?NI@K?@E> 

 >I<<EK<EK8:C<J #FN:FLC;$K<CCN?<K?<I@KN8J89C<>> FI8IL9<a

`2FNEFFE<cJ<M<I=FLE;8EF9A<:K that D@O<;LGaJ8PJ.LJ8E`9LK$>L<JJN<c;K8B<@KKFK?<JFIK@E> J:8EE<Ia

`#FN;F<JK?<JFIK@E>J:8EE<INFIBaPFL@EHL@I<`3eI8PJ (8>E<K@:I<JFE8E:<@D8>@E> !8JKE<LKIFEKI8EJD@JJ@FEJG<:KIFJ:FGPa

`$cDKFC;@KNFIBJ9P8P<JcJ-LC<9LK$;FEcKHL@K<LE;<IJK8E;?FNaJ8PJ.LJ8E `$C@B<KFJ8P@KK?FL>?8P<J8P<J8P<J8P<J8P<Ja

`2?8K;F<JK?<JFIK@E>J:8EE<I tell PFLa

`$KK<CCJPFLN?<K?<IKFGLKK?<F9A<:K@EKFK?<9C<>> 

-%&6+-!,'/'*+

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 215/309

9@EFIK?<IL9<9@E /?8KcJN?P@KcJ:8CC<;8JFIK@E> J:8EE<Ia

 KK?@JGF@EKPFL=8CCJ@C<EK

`$E:@;<EK8CCPa.LJ8EJ8PJ:8JL8CCP`@KD8P@EK<I<JKPFLKFBEFNK?8K9C<>>J:FEK8@EJD8CCEL>><KJF=M8E8;@LDFI<8E;IL9<J:FEK8@EJ?I<;JF=G8CC8;@LD9FK?F= 

 N?@:?8I<LJ<=LC@E;LJKI@8CCPa`.LJ8EPFL8I<GLI<<M@Ca`/?8EBPFLa

.F EFN @K J<<DJ N<cM< ;@J:FM<I<; K?< ?<8IK 8E; <JJ<E:< F= 9C<>e >E<JJ 8 9C<>> @J 8E F9A<:K K?8K :FEK8@EJ 8 EL>><K F= M8E8;@LD FI<

.LI=8:< :?8I8:K<I@JK@:J C@B< 9CL< :FCFI 8E; =LII<;E<JJ ;F EFK de-termine N?<K?<I 8E F9A<:K @J 8 9C<>> JLI=8:< :?8I8:K<I@JK@:J FECP D8KK<I 9<:8LJ< K?<P ?<CG PFL infer N?<K?<I 8E F9A<:K @J 8 9C<>> K?8K@JN?<K?<IK?<F9A<:K:FEK8@EJM8E8;@LD

FEK8@E@E> M8E8;@LD @J 8 E<:<JJ8IP 8E; JL==@:@<EK ;<=@E@K@FE 8CC9C<>>J :FEK8@E M8E8;@LD 8E; <M<IPK?@E> K?8K :FEK8@EJ M8E8;@LD @J 8 9C<>> `9C<>>a @J ALJK 8 J?FIK?8E; N8P F= J8P@E> `M8E8;@LDe:FEK8@Ee@E>F9A<:Ka -@>?K

)FK JF =8JK J8PJ .LJ8E IFLE; n F= 9C<>>J :FEK8@E M8E8e;@LD 9LK n :FEK8@E G8CC8;@LD @EJK<8; /F 9< GI<:@J< h.LJ8E:FEK@EL<Ji 8IFLE; n F= 9CL< <>>eJ?8G<; =LII<; =C<O@9C< FG8HL<F9A<:KJ :FEK8@E M8E8;@LD !FI LELJL8C 9C<>>J @K D8P 9< 8 ;@==<I<EKG<I:<EK8>< n F= GLIGC< 9C<>>J :FEK8@E M8E8;@LD n F= ?8I;9C<>>J:FEK8@EM8E8;@LD<K:

)FN JLGGFJ< PFL =@E; 8 9CL< <>>eJ?8G<; =LII<; =C<O@9C< FG8HL<F9A<:K 8E FI;@E8IP 9C<>> @E <M<IP M@J@9C< N8P 8E; ALJK =FI B@:BJ

 PFL K8B< @K KF K?< JFIK@E> J:8EE<I 8E; K?< J:8EE<I J8PJ `G8CC8;@e

LDagK?@J@JFE<F=K?<I8I<n $J@K89C<>> K =@IJK PFL D@>?K 8EJN<I K?8K J@E:< PFL @EK<E; KF K?IFN K?@J

F9A<:K @E K?< IL9< 9@E PFL D@>?K 8J N<CC :8CC @K 8 `IL9<a #FN<M<I@K KLIEJ FLK K?8K 8CDFJK 8CC 9C<>>J @= PFL JN@K:? F== K?< C@>?KJ >CFN =8@EKCP @E K?< ;8IB N?@C< 8CDFJK 8CC IL9<J ;F EFK >CFN @E K?< ;8IB

 E; K?< G<I:<EK8>< F= 9C<>>J K?8K >CFN @E K?< ;8IB @J EFK J@>E@=@e:8EKCP ;@==<I<EK =FI 9CL< <>>eJ?8G<; =LII<; =C<O@9C< FG8HL< F9A<:KJK?8K :FEK8@E G8CC8;@LD @EJK<8; F= M8E8;@LD /?LJ @= PFL N8EK KF

 >L<JJ N?<K?<I K?< F9A<:K >CFNJ C@B< 8 9C<>> FI I<D8@EJ ;8IB C@B< 8 

IL9<PFLJ?FLC;>L<JJK?8K@K>CFNJC@B<89C<>>

!+-!+)-*!+

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 216/309

.F@JK?<F9A<:K really 89C<>>FI8IL9<

*E FE< ?8E; PFLcCC K?IFN K?< F9A<:K @E K?< IL9< 9@E EF D8KK<I N?8K <CJ< PFL C<8IE *E K?< FK?<I ?8E; @= K?<I< 8I< 8EP LEBEFNE

:?8I8:K<I@JK@:J F= K?< F9A<:K PFL E<<; KF @E=<I PFLcCC @E=<I K?<D 8J@= K?< F9A<:K N<I< 8 9C<>> EFK 8 IL9<g>IFLG @K @EKF K?< J@D@C8I@KP :CLJK<I F= 9CL< <>>eJ?8G<; =LII<; =C<O@9C< FG8HL< K?@E>J 8E; EFKK?< J@D@C8I@KP :CLJK<I F= I<; :L9<eJ?8G<; JDFFK? ?8I; KI8EJCL:<EKK?@E>J

/?< HL<JK@FE `$J K?@J F9A<:K 8 9C<>>a D8P JK8E; @E =FI ;@==<I<EKHL<I@<JFE;@==<I<EKF::8J@FEJ

$= @K N<I<EcK JK8E;@E> @E =FI some HL<IP PFLc; ?8M< EF I<8JFE KF:8I<

$J 8K?<@JD 8 `I<C@>@FEa $J KI8EJ?LD8E@JD 8 `:LCKa +<FGC< N?F 8I>L< K?8K 8K?<@JD @J 8 I<C@>@FE `9<:8LJ< @K JK8K<J 9<C@<=J 89FLK"F;a 8I< I<8CCP KIP@E> KF 8I>L< h$ K?@EBi K?8K K?< I<8JFE@E> D<K?F;JLJ<; @E 8K?<@JD 8I< FE 8 G8I N@K? K?< I<8JFE@E> D<K?F;J LJ<; @EI<C@>@FE FI K?8K 8K?<@JD @J EF J8=<I K?8E I<C@>@FE @E K<IDJ F= K?<GIF989@C@KP F= :8LJ8CCP <E><E;<I@E> M@FC<E:< <K:_ 2?8KcJ I<8CCP 8KJK8B< @J 8E 8K?<@JKcJ :C8@D F= JL9JK8EK@8C ;@==<I<E:< 8E; JLG<I@FI@KP I<C8K@M< KF I<C@>@FE N?@:? K?< I<C@>@FLJ G<IJFE @J KIP@E> KF I<A<:K 9P 

;<EP@E>K?<;@==<I<E:<I8K?<IK?8EK?<JLG<I@FI@KPhiLK K?8KcJ EFK K?< 8 GI@FI@ @II8K@FE8C G8IK /?< 8 GI@FI@ @II8K@FE8C

G8IK @J N?<I< @E K?< :FLIJ< F= K?< 8I>LD<EK JFD<FE< GLCCJ FLK 8 ;@:K@FE8IP 8E; CFFBJ LG K?< ;<=@E@K@FE F= `8K?<@JDa FI `I<C@>@FEahE; P<J @KcJ ALJK 8J J@CCP N?<K?<I 8E 8K?<@JK FI I<C@>@FE@JK ;F<J @Ki#FN :FLC; 8 ;@:K@FE8IP possibly ;<:@;< N?<K?<I 8E <DG@I@:8C :CLJK<IF= 8K?<@JKJ @J I<8CCP JL9JK8EK@8CCP ;@==<I<EK =IFD 8E <DG@I@:8C :CLJeK<I F= K?<FCF>@8EJ #FN :8E I<8C@KP M8IP N@K? K?< D<8E@E> F= 8 

 NFI; /?< GF@EKJ @E K?@E>JG8:< ;FEcK DFM< 8IFLE; N?<E N< I<e

;I8N89FLE;8IPLK G<FGC< F=K<E ;FEcK realize K?8K K?<@I 8I>LD<EK 89FLK N?<I<

KF ;I8N 8 ;<=@E@K@FE8C 9FLE;8IP @J I<8CCP 8 ;@JGLK< FM<I N?<K?<I KF@E=<I 8 :?8I8:K<I@JK@: J?8I<; 9P DFJK K?@E>J @EJ@;< 8E <DG@I@:8C :CLJeK<I_

#<E:<K?<G?I8J<`;@J>L@J<;HL<IPa

+8><b0E:I@K@:8C.LG<I:I@K@:8C@KPc +8><bLCK@J?FLEK<I:LCK@J?E<JJc

-%&6+-!,'/'*+

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 217/309

9. Neural Categories

Followup to @J>L@J<;,L<I@<J

$E @J>L@J<; ,L<I@<J $ K8CB<; 89FLK 8 :C8JJ@=@:8K@FE K8JB F= `9C<>>Ja 8E; `IL9<Ja /?< KPG@:8C 9C<>> @J 9CL< <>>eJ?8G<; =LII<;=C<O@9C< FG8HL< >CFNJ @E K?< ;8IB 8E; :FEK8@EJ M8E8;@LD /?<KPG@:8C IL9< @J I<; :L9<eJ?8G<; JDFFK? ?8I; KI8EJCL:<EK LE>CFNe@E> 8E; :FEK8@EJ G8CC8;@LD !FI K?< J8B< F= J@DGC@:@KP C<K LJ =FI><KK?< :?8I8:K<I@JK@:J F= =C<O@9@C@KP?8I;E<JJ 8E; FG8HL<E<JJKI8EJCLe:<E:P /?@J C<8M<J =@M< ;@D<EJ@FEJ @E K?@E>JG8:< FCFI J?8G<K<OKLI<CLD@E8E:<8E;@EK<I@FI

.LGGFJ< $ N8EK KF :I<8K< 8E IK@=@:@8C )<LI8C )<KNFIB h))iKF GI<;@:K LEF9J<IM<; 9C<>> :?8I8:K<I@JK@:J =IFD F9J<IM<; 9C<>> :?8I8:K<I@JK@:J E; JLGGFJ< $cD =8@ICP E8@M< 89FLK ))J $cM<I<8; <O:@K<; GFGLC8I J:@<E:< 9FFBJ 89FLK ?FN E<LI8C E<KNFIBJ 8I<;@JKI@9LK<; <D<I><EK 8E; G8I8CC<C just like the human brain!!  9LK $:8EcK ;<I@M< K?< ;@==<I<EK@8C <HL8K@FEJ =FI >I8;@<EK ;<J:<EK @E 8 EFEeI<:LII<EK DLCK@C8P<I E<KNFIB N@K? J@>DF@; LE@KJ hN?@:? @J 8:KL8CCP 8 CFK<8J@<IK?8E@KJFLE;Ji

/?<E $ D@>?K ;<J@>E 8 E<LI8C E<KNFIB K?8K CFFBJ JFD<K?@E> C@B<

K?@J

http://lesswrong.com/lw/nn/neural_categories/  +8><b@J>L@J<;,L<I@<Jc

+8><b@J>L@J<;,L<I@<Jc +8><b/?<CLJK<I.KIL:KLI<F=/?@E>JG8:<c

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 218/309

C<>>7

)<KNFIB @J =FI :C8JJ@=P@E> 9C<>>J 8E; IL9<J LK J@E:< `9C<>>a@J 8E LE=8D@C@8I 8E; JPEK?<K@: :FE:<GK $cM< 8CJF @E:CL;<; 8 J@D@C8I)<KNFIB 9 =FI ;@JK@E>L@J?@E> ?LD8EJ =IFD .G8:< (FEJK<IJ N@K?

@EGLK =IFD I@JKFKC< h`CC D<E 8I< DFIK8Cai 8E; +C8KFcJ :8;<DP h`=<8K?<IC<JJ9@G<;N@K?9IF8;E8@CJai

  E<LI8C E<KNFIB E<<;J 8 C<8IE@E> ILC< /?< F9M@FLJ @;<8 @JK?8K N?<E KNF EF;<J 8I< F=K<E 8:K@M< 8K K?< J8D< K@D< N< J?FLC;JKI<E>K?<E K?< :FEE<:K@FE 9<KN<<E K?<DgK?@J @J FE< F= K?< =@IJKILC<J <M<I GIFGFJ<; =FI KI8@E@E> 8 E<LI8C E<KNFIB BEFNE 8J #<99cJ-LC<

/?LJ @= PFL F=K<E J8N K?@E>J K?8K N<I< 9FK? 9CL< 8E;=LII<;gK?LJ J@DLCK8E<FLJCP 8:K@M8K@E> K?< `:FCFIa EF;< @E K?<

JK8K< 8E; K?< `K<OKLI<a EF;< @E K?< JK8K<gK?< :FEE<:K@FE NFLC;JKI<E>K?<E 9<KN<<E :FCFI 8E; K<OKLI< JF K?8K :FCFIJ 8:K@M8K<; K<OKLI<J 8E; M@:< M<IJ8 $= PFL J8N K?@E>J K?8K N<I< 9CL< 8E; <>>eJ?8G<; 8E; M8E8;@LDe:FEK8@E@E> K?8K NFLC; JKI<E>K?<E GFJ@K@M<DLKL8C:FEE<:K@FEJ9<KN<<E:FCFI8E;J?8G<8E;@EK<I@FI

'<KcJ J8P PFLcM< 8CI<8;P J<<E GC<EKP F= 9C<>>J 8E; IL9<J :FD< F== K?< :FEM<PFI 9<CK LK EFN PFL J<< JFD<K?@E> K?8KcJ =LII<; <>>eJ?8G<; 8E;g>8JGgI<;;@J? GLIGC< hN?@:? N<cCC DF;<C 8J 8 `:FCFIa

-%&6+-!,'/'*+

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 219/309

8:K@M8K@FE C<M<C F= ei 4FL ?8M<EcK P<K K<JK<; K?< CLD@E8E:< FIK?<@EK<I@FI 2?8KKFGI<;@:KN?8KKFGI<;@:K

2?8K ?8GG<EJ K?<E @J K?8K K?< 8:K@M8K@FE C<M<CJ @E )<KNFIB

9FLE:< 8IFLE; 8 9@K +FJ@K@M< 8:K@M8K@FE =CFNJ CLD@E8E:< =IFDJ?8G< E<>8K@M< 8:K@M8K@FE =CFNJ KF @EK<I@FI =IFD :FCFI E<>8K@M<8:K@M8K@FE =CFNJ =IFD @EK<I@FI KF CLD@E8E:<_ *= :FLIJ< 8CC K?<J<D<JJ8><J 8I< G8JJ<; @E parallel!! 8E; asynchronously!! ALJK C@B< K?< ?LeD8E9I8@E_

!@E8CCP )<KNFIB J<KKC<J @EKF 8 JK89C< JK8K< N?@:? ?8J ?@>? GFJe@K@M< 8:K@M8K@FE =FI `CLD@E8E:<a 8E; `@EK<I@FIa /?< E<KNFIB D8P 9< J8@; KF `<OG<:Ka hK?FL>? @K ?8J EFK P<K J<<Ei K?8K K?< F9A<:K N@CC

 >CFN@EK?<;8IB8E;K?8K@K:FEK8@EJM8E8;@LD

 E; CF )<KNFIB <O?@9@KJ K?@J 9<?8M@FI <M<E K?FL>? K?<I<cJ EF<OGC@:@K EF;< K?8K J8PJ N?<K?<I K?< F9A<:K @J 8 9C<>> FI EFK /?<AL;>D<EK @J implicit in the whole network!!  C<>>E<JJ @J 8E attractor!! 

 N?@:? 8I@J<J 8J K?< I<JLCK F= emergent behavior!! =IFD K?< distributed!! C<8IE@E>ILC<

)FN @E I<8C C@=< K?@J B@E; F= E<KNFIB ;<J@>Eg?FN<M<I =8;;@J?

@K D8P JFLE;gILEJ @EKF all sorts F= GIF9C<DJ -<:LII<EK E<KNFIBJ;FEcK 8CN8PJ J<KKC< I@>?K 8N8P /?<P :8E FJ:@CC8K< FI <O?@9@K :?8FK@:

9<?8M@FI FI ALJK K8B< 8 M<IP CFE> K@D< KF J<KKC< ;FNE /?@J @J 8 8;/?@E> N?<E PFL J<< JFD<K?@E> 9@> 8E; P<CCFN 8E; JKI@G<; 8E; PFL?8M< KF N8@K =@M< D@ELK<J =FI PFLI ;@JKI@9LK<; E<LI8C E<KNFIB KF J<KeKC< @EKF K?< `K@><Ia 8KKI8:KFI JPE:?IFEFLJ 8E; G8I8CC<C @K D8P 9<9LK@KcJEFKI<8CeK@D<

 E; K?<I< 8I< FK?<I GIF9C<DJ C@B< ;FL9C<e:FLEK@E> K?< <M@e;<E:<  N?<E D<JJ8><J 9FLE:< 98:B 8E; =FIK? $= PFL JLJG<:K K?8K8E F9A<:K >CFNJ @E K?< ;8IB PFLI JLJG@:@FE N@CC 8:K@M8K< 9<C@<= K?8KK?< F9A<:K :FEK8@EJ M8E8;@LD N?@:? @E KLIE N@CC 8:K@M8K< 9<C@<= K?8KK?<F9A<:K>CFNJ@EK?<;8IB

+CLJ @= PFL KIP KF J:8C< LG K?< )<KNFIB ;<J@>E @K I<HL@I<J

*h) i:FEE<:K@FEJN?<I<)@JK?<KFK8CELD9<IF=F9J<IM89C<J

.FN?8KD@>?K9<8DFI<I<8C@JK@:E<LI8CE<KNFIB;<J@>E

http://thedailywtf.com/Articles/No,_We_Need_a_Neural_Network.aspx +8><b!8B<8LJ8C@KPc

 &-*$,'*!+

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 220/309

C<>>

$E K?@J E<KNFIB 8 N8M< F= 8:K@M8K@FE :FEM<I><J FE K?< :<EKI8C EF;<=IFD 8EP :C8DG<; hF9J<IM<;i EF;<J 8E; K?<E JLI><J 98:B FLK 8>8@EKF 8EP LE:C8DG<; hLEF9J<IM<;i EF;<J 2?@:? D<8EJ N< :8E :FDeGLK< K?< 8EJN<I @E FE< JK<G I8K?<I K?8E N8@K@E> =FI K?< E<KNFIB KFJ<KKC<g8E @DGFIK8EK I<HL@I<D<EK @E 9@FCF>P N?<E K?< E<LIFEJ FECP ILE 8K #Q E; K?< E<KNFIB 8I:?@K<:KLI< J:8C<J 8J *h)i I8K?<I

K?8E*h) i

 ;D@KK<;CP K?<I< 8I< JFD< K?@E>J PFL :8E EFK@:< DFI< <8J@CP  N@K? K?< =@IJK E<KNFIB 8I:?@K<:KLI< K?8E K?< J<:FE; )<KNFIB ?8J 8 ;@I<:K :FEE<:K@FE 9<KN<<E <M<IP KNF EF;<J .F @= I<; F9A<:KJ never  >CFN @E K?< ;8IB 9LK I<; =LII<; F9A<:KJ LJL8CCP ?8M< K?<FK?<I 9C<>> :?8I8:K<I@JK@:J C@B< <>>eJ?8G< 8E; M8E8;@LD )<KNFIB :8E <8J@CP I<GI<J<EK K?@J @K ALJK K8B<J 8 M<IP JKIFE> ;@I<:K E<>8K@M<

:FEE<:K@FE =IFD :FCFI KF CLD@E8E:< 9LK DFI< GFN<I=LC GFJ@K@M<:FEE<:K@FEJ=IFDK<OKLI<KF8CCFK?<IEF;<J<O:<GKCLD@E8E:<

)FI @J K?@J 8 `JG<:@8C <O:<GK@FEa KF K?< ><E<I8C ILC< K?8K 9C<>>J >CFNgI<D<D9<I @E )<KNFIB K?<I< @J EF LE@K K?8K I<GI<J<EKJ9C<>>eE<JJ 9C<>>eE<JJ <D<I><J 8J 8E 8KKI8:KFI @E K?< ;@JKI@9LK<;E<KNFIB

.F P<J K?FJ< ) :FEE<:K@FEJ N<I< 9LP@E> LJ JFD<K?@E> LKEFK M<IP DL:? )<KNFIB @J EFK more LJ<=LC FE DFJK I<8CeNFIC;

-%&6+-!,'/'*+

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 221/309

GIF9C<DJ N?<I< PFL I8I<CP =@E; 8E 8E@D8C JKL:B ?8C=N8P 9<KN<<E9<@E>8:8K8E;8;F>

h/?<I< 8I< 8CJF =8:KJ K?8K PFL :8EcK <8J@CP I<GI<J<EK @E )<KNFIB

or )<KNFIB '<KcJ J8P J<8e9CL< :FCFI 8E; JG?<IF@; J?8G< N?<E=FLE; KF><K?<I 8CN8PJ @E;@:8K< K?< GI<J<E:< F= G8CC8;@LD 9LK

 N?<E =FLE; @E;@M@;L8CCP N@K?FLK K?< FK?<I K?<P 8I< <8:? M<IP JKIFE> <M@;<E:< =FI M8E8;@LD /?@J @J ?8I; KF I<GI<J<EK @E <@K?<I8I:?@K<:KLI< N@K?FLK <OKI8 EF;<J FK? )<KNFIB 8E; )<KNFIB <D9F;P @DGC@:@K 8JJLDGK@FEJ 89FLK N?8K B@E; F= <EM@IFED<EK8CJKIL:KLI< @J C@B<CP KF <O@JK K?< 89@C@KP KF I<8; K?@J F== @J N?8K J<G8eI8K<JK?<8;LCKJ=IFDK?<989<J@ED8:?@E<C<8IE@E>i

(8B< EF D@JK8B< )<@K?<I )<KNFIB EFI )<KNFIB 8I< 9@e

FCF>@:8CCP I<8C@JK@: But  @K JK@CC J<<DJ C@B< 8 =8@I >L<JJ K?8K ?FN<M<IK?< 9I8@E I<8CCP NFIBJ @K @J @E JFD< J<EJ< :CFJ<I KF )<KNFIB K?8E)<KNFIB !8JK :?<8G J:8C89C< NFIBJ N<CC KF ;@JK@E>L@J? ;F>J 8E;:8KJ E8KLI8C J<C<:K@FE >F<J =FI K?8K JFIK F= K?@E> C@B< N8K<I ILEE@E> ;FNE8=@KE<JJC8E;J:8G<

$K J<<DJ C@B< 8E FI;@E8IP <EFL>? K8JB KF :C8JJ@=P F9A<:KJ 8J <@K?<I9C<>>J FI IL9<J KFJJ@E> K?<D @EKF K?< 8GGIFGI@8K< 9@E LK NFLC;

 PFLEFK@:<@=J<8e9CL<F9A<:KJE<M<I>CFN<;@EK?<;8IB

(8P9< @= JFD<FE< GI<J<EK<; PFL N@K? KN<EKP F9A<:KJ K?8K N<I<8C@B< FECP @E 9<@E> J<8e9CL< 8E; K?<E JN@K:?<; F== K?< C@>?K 8E;EFE< F= K?< F9A<:KJ >CFN<; $= PFL >FK ?@K FM<I K?< ?<8; N@K? @K@E FK?<I NFI;J +<I?8GJ 9P GI<J<EK@E> PFL N@K? 8CC K?<J< J<8e9CL<F9A<:KJ @E 8 >IFLG PFLI 9I8@E =FIDJ 8 E<N JL9:8K<>FIP 8E; :8E ;<eK<:K K?< `;F<JEcK >CFNa :?8I8:K<I@JK@: N@K?@E K?8K JL9:8K<>FIP LK

 PFL GIF989CP NFLC;EcK EFK@:< @= K?< J<8e9CL< F9A<:KJ N<I< J:8KK<I<;8DFE> 8 ?LE;I<; FK?<I 9C<>>J 8E; IL9<J $K NFLC;EcK 9< easy FI in-tuitive KF EFK@:< K?< N8P K?8K ;@JK@E>L@J?@E> :8KJ 8E; ;F>J @J <8JP 

8E;@EKL@K@M<*I `.F:I8K<J @J ?LD8E 8CC ?LD8EJ 8I< DFIK8C K?<I<=FI<

.F:I8K<J @J DFIK8Ca #FN ;@; I@JKFKC< BEFN K?8K .F:I8K<J N8J ?LeD8E 2<CC .F:I8K<J ?8; EF =<8K?<IJ 8E; 9IF8; E8@CJ 8E; N8CB<;LGI@>?K 8E; JGFB< "I<<B 8E; N<CC N8J ><E<I8CCP J?8G<; C@B< 8 ?LeD8E 8E; 8:K<; C@B< FE< .F K?< 9I8@E ;<:@;<J FE:< 8E; =FI 8CC K?8K.F:I8K<J @J ?LD8E 8E; =IFD K?<I< @E=<IJ K?8K .F:I8K<J @J DFIK8CC@B< 8CC FK?<I ?LD8EJ K?LJ P<K F9J<IM<; $K ;F<JEcK J<<D <8JP FI@EKL@K@M< KF 8JB ?FN DL:? N<8I@E> :CFK?<J 8J FGGFJ<; KF LJ@E> C8Ee

 &-*$,'*!+

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 222/309

 >L8>< @J 8JJF:@8K<; N@K? DFIK8C@KP %LJK `K?@E>J K?8K N<8I :CFK?<J8E;LJ<C8E>L8><8I<?LD8Ea8E;`?LD8EJ8I<DFIK8Ca

 I< K?<I< 9@8J<J 8JJF:@8K<; N@K? KIP@E> KF :C8JJ@=P K?@E>J @EKF

:8K<>FI@<J FE:< 8E; =FI 8CC *= :FLIJ< K?<I< 8I< .<< <> LCK@J?FLEK<I:LCK@J?E<JJ

/F9<:FEK@EL<;_

+8><bLCK@J?FLEK<I:LCK@J?E<JJc

-%&6+-!,'/'*+

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 223/309

10. How An Algorithm Feels From Inside

Followup to )<LI8C8K<>FI@<J

`$= 8 KI<< =8CCJ @E K?< =FI<JK 8E; EF FE< ?<8IJ @K ;F<J @K D8B< 8 JFLE;a $ I<D<D9<I J<<@E> 8E 8:KL8C 8I>LD<EK ><K JK8IK<; FE K?@JJL9A<:Kg8 =LCCP E8@M< 8I>LD<EK K?8K N<EK EFN?<I< E<8I <IB<C<P8EJL9A<:K@M@JD %LJK

`$KD8B<J8JFLE;ALJKC@B<8EPFK?<I=8CC@E>KI<<a`LK?FN:8EK?<I<9<8JFLE;K?8KEFFE<?<8IJa

/?< JK8E;8I; I8K@FE8C@JK M@<N NFLC; 9< K?8K K?< =@IJK G<IJFE @JJG<8B@E> 8J @= `JFLE;a D<8EJ 8:FLJK@: M@9I8K@FEJ @E K?< 8@I K?< J<:eFE; G<IJFE @J JG<8B@E> 8J @= `JFLE;a D<8EJ 8E 8L;@KFIP <OG<I@<E:<@E 8 9I8@E $= PFL 8JB `I< K?<I< 8:FLJK@: M@9I8K@FEJa FI `I< K?<I<8L;@KFIP <OG<I@<E:<Ja K?< 8EJN<I @J 8K FE:< F9M@FLJ E; JF K?<8I>LD<EK@JI<8CCP89FLKK?<;<=@E@K@FEF=K?<NFI;`JFLE;a

$ K?@EB K?< JK8E;8I; 8E8CPJ@J @J <JJ<EK@8CCP :FII<:K .F C<KcJ 8:e:<GK K?8K 8J 8 GI<D@J< 8E; 8JB 2?P ;F G<FGC< ><K @EKF JL:? 8E8I>LD<EK 2?8KcJK?<LE;<ICP@E>GJP:?FCF>P

  B<P @;<8 F= K?< ?<LI@JK@:J 8E; 9@8J<J GIF>I8D @J K?8K D@JK8B<J8I< F=K<E DFI< I<M<8C@E> F= :F>E@K@FE K?8E :FII<:K 8EJN<IJ "<KeK@E> @EKF 8 ?<8K<; ;@JGLK< 89FLK N?<K?<I @= 8 KI<< =8CCJ @E 8 ;<J<IK<;=FI<JK@KD8B<J8JFLE;@JKI8;@K@FE8CCP:FEJ@;<I<;8D@JK8B<

.FN?8KB@E;F=D@E;;<J@>E:FII<JGFE;JKFK?8K<IIFI

$E @J>L@J<; ,L<I@<J $ @EKIF;L:<; K?< 9C<>>IL9< :C8JJ@=@:8K@FEK8JB @E N?@:? .LJ8E K?< .<E@FI .FIK<I <OGC8@EJ K?8K PFLI AF9 @J KFJFIK F9A<:KJ :FD@E> F== 8 :FEM<PFI 9<CK GLKK@E> K?< 9CL< <>>J FI

`9C<>>Ja @EKF FE< 9@E 8E; K?< I<; :L9<J FI `IL9<Ja @EKF K?< IL9<9@E /?@J @K KLIEJ FLK @J 9<:8LJ< 9C<>>J :FEK8@E JD8CC EL>><KJ F=  M8E8;@LD FI< 8E; IL9<J :FEK8@E JD8CC J?I<;J F= G8CC8;@LD 9FK? F=  N?@:?8I<LJ<=LC@E;LJKI@8CCP

O:<GK K?8K 8IFLE; n F= 9CL< <>>eJ?8G<; F9A<:KJ :FEK8@E G8CeC8;@LD @EJK<8; .F @= PFL =@E; 8 9CL< <>>eJ?8G<; K?@E> K?8K :FEK8@EJ

http://lesswrong.com/lw/no/how_an_algorithm_feels_from_inside/ 

+8><b)<LI8C8K<>FI@<Jc +8><b@J>L@J<;,L<I@<Jc

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 224/309

G8CC8;@LD J?FLC; PFL :8CC @K 8 `IL9<a @EJK<8; 4FLcI< >F@E> KF GLK@K@EK?<IL9<9@EgN?PEFK:8CC@K8`IL9<a

LK N?<E PFL JN@K:? F== K?< C@>?K E<8ICP 8CC 9C<>>J >CFN =8@EKCP 

@E K?< ;8IB E; 9CL< <>>eJ?8G<; F9A<:KJ K?8K :FEK8@E G8CC8;@LD8I< ALJK 8J C@B<CP KF >CFN @E K?< ;8IB 8J 8EP FK?<I 9CL< <>>eJ?8G<;F9A<:K

.F @= PFL =@E; 8 9CL< <>>eJ?8G<; F9A<:K K?8K :FEK8@EJ G8CC8;@LD8E; PFL 8JB `$J @K 8 9C<>>a K?< 8EJN<I ;<G<E;J FE N?8K PFL ?8M< KF;F N@K? K?< 8EJN<I $= PFL 8JB `2?@:? 9@E ;F<J K?< F9A<:K >F @EaK?<E PFL :?FFJ< 8J @= K?< F9A<:K @J 8 IL9< LK @= PFL 8JB `$= $ KLIEF== K?< C@>?K N@CC @K >CFNa PFL GI<;@:K 8J @= K?< F9A<:K @J 8 9C<>>$E FE< :8J< K?< HL<JK@FE `$J @K 8 9C<>>a JK8E;J @E =FI K?< ;@J>L@J<;

HL<IP  `2?@:? 9@E ;F<J @K >F @Ea $E K?< FK?<I :8J< K?< HL<JK@FE`$J @K 8 9C<>>a JK8E;J @E =FI K?< ;@J>L@J<; HL<IP  `2@CC @K >CFN @EK?<;8IBa

)FN JLGGFJ< K?8K PFL ?8M< 8E F9A<:K K?8K @J 9CL< 8E; <>>eJ?8G<; 8E; :FEK8@EJ G8CC8;@LD 8E; PFL ?8M< 8CI<8;P F9J<IM<; K?8K@K@J=LII<;=C<O@9C<FG8HL<8E;>CFNJ@EK?<;8IB

/?@J 8EJN<IJ every HL<IP F9J<IM<J <M<IP F9J<IM89C< @EKIF;L:<;/?<I<cJEFK?@E>C<=K=FI8;@J>L@J<;HL<IPKFJK8E; for.

.F N?P D@>?K JFD<FE< =<<C 8E @DGLCJ< KF >F FE 8I>L@E> N?<K?<IK?<F9A<:K@J really 89C<>>

+8><b@J>L@J<;,L<I@<Jc +8><b@J>L@J<;,L<I@<Jc

-%&6+-!,'/'*+

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 225/309

C<>>

/?@J ;@8>I8D =IFD )<LI8C 8K<>FI@<J J?FNJ KNF ;@==<I<EK E<LeI8C E<KNFIBJ K?8K D@>?K 9< LJ<; KF 8EJN<I HL<JK@FEJ 89FLK 9C<>>J8E; IL9<J )<KNFIB ?8J 8 ELD9<I F= ;@J8;M8EK8><JgJL:? 8J

GFK<EK@8CCP FJ:@CC8K@E>:?8FK@: 9<?8M@FI FI I<HL@I@E> *h)

 i :FEeE<:K@FEJg9LK )<KNFIB cJ JKIL:KLI< ;F<J ?8M< FE< D8AFI 8;M8EeK8>< FM<I )<KNFIB M<IP LE@K @E K?< E<KNFIB :FII<JGFE;J KF 8 K<JK89C< HL<IP $= PFL F9J<IM< <M<IP F9J<IM89C< :C8DG@E> <M<IP M8CeL<K?<I<8I<EFLE@KJ@EK?<E<KNFIBC<=KFM<I

)<KNFIB ?FN<M<I @J 8 =8I 9<KK<I :8E;@;8K< =FI 9<@E> JFD<eK?@E> M8>L<CP C@B< ?FN K?< ?LD8E 9I8@E NFIBJ $KcJ =8JK :?<8GJ:8C89C<g8E; ?8J 8E <OKI8 ;8E>C@E> LE@K @E K?< :<EK<I N?FJ< 8:K@e

 M8K@FE :8E JK@CC M8IP <M<E 8=K<I N<cM< F9J<IM<; <M<IP J@E>C< FE< F= 

K?<JLIIFLE;@E>EF;<J2?@:? @J KF J8P K?8K <M<E 8=K<I PFL BEFN N?<K?<I 8E F9A<:K

@J 9CL< FI I<; <>> FI :L9< =LII<; FI JDFFK? 9I@>?K FI ;8IB 8E; N?<K?<I @K :FEK8@EJ M8E8;@LD FI G8CC8;@LD @K feels C@B< K?<I<cJ 8 C<=KeFM<ILE8EJN<I<;HL<JK@FE But is it really a blegg? 

0JL8CCP @E FLI ;8@CP <OG<I@<E:< 8:FLJK@: M@9I8K@FEJ 8E; 8L;@KFIP <OG<I@<E:< >F KF><K?<I LK 8 KI<< =8CC@E> @E 8 ;<J<IK<; =FI<JK LEe9LE;C<J K?@J :FDDFE 8JJF:@8K@FE E; <M<E 8=K<I PFL BEFN K?8K

+8><b)<LI8C8K<>FI@<Jc

'/ &$'*!,%$+*'%! &+!

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 226/309

K?< =8CC@E> KI<< :I<8K<J 8:FLJK@: M@9I8K@FEJ 9LK EFK 8L;@KFIP <OG<I@e<E:<@K feels C@B<K?<I<cJ8C<=KFM<IHL<JK@FE Did it make a sound? 

2< BEFN N?<I< +CLKF @J 8E; N?<I< @KcJ >F@E> N< BEFN +CLKFcJJ?8G<8E;+CLKFcJD8JJg9LK@J@K8GC8E<K

)FN I<D<D9<I 2?<E PFL CFFB 8K )<KNFIB 8J $cM< C8@;@K FLK ?<I< PFLcI< J<<@E> K?< 8C>FI@K?D =IFD K?< FLKJ@;< +<FGC<;FEcK K?@EB KF K?<DJ<CM<J `.?FLC; K?< :<EKI8C LE@K =@I< FI EFKa 8EP DFI< K?8E PFL K?@EB `.?FLC; E<LIFE @E DP M@JL8C:FIK<O=@I<FIEFKa

$K K8B<J 8 ;<C@9<I8K< <==FIK KF M@JL8C@Q< PFLI 9I8@E =IFD K?< FLKeJ@;<g8E; K?<E PFL JK@CC ;FEcK J<< PFLI 8:KL8C 9I8@E PFL @D8>@E<

 N?8K PFL think @J K?<I< ?FG<=LCCP 98J<; FE J:@<E:< 9LK I<>8I;C<JJ PFL ;FEcK ?8M< 8EP ;@I<:K 8::<JJ KF E<LI8C E<KNFIB JKIL:KLI<J =IFD@EKIFJG<:K@FE /?8KcJ N?P K?< 8E:@<EK "I<<BJ ;@;EcK @EM<EK :FDGLeK8K@FE8CE<LIFJ:@<E:<

2?<E PFL CFFB 8K )<KNFIB PFL 8I< J<<@E> =IFD K?< outside;9LK K?< N8P K?8K E<LI8C E<KNFIB JKIL:KLI< =<<CJ =IFD K?< inside, @= 

 PFL PFLIJ<C= are 8 9I8@E ILEE@E> K?8K 8C>FI@K?D @J K?8K <M<E 8=K<I PFL BEFN <M<IP :?8I8:K<I@JK@: F= K?< F9A<:K PFL JK@CC =@E; PFLIJ<C= 

 NFE;<I@E> `LK@J@K89C<>>FIEFKa/?@J @J 8 >I<8K >8G KF :IFJJ 8E; $cM< J<<E @K JKFG G<FGC< @E K?<@IKI8:BJ <:8LJ< N< ;FEcK @EJK@E:K@M<CP J<< FLI @EKL@K@FEJ 8J `@EKLe@K@FEJa N< ALJK J<< K?<D 8J K?< NFIC; 2?<E PFL CFFB 8K 8 >I<<E:LG PFL ;FEcK K?@EB F= PFLIJ<C= 8J J<<@E> 8 G@:KLI< I<:FEJKIL:K<; @E

 PFLI M@JL8C :FIK<Og8CK?FL>? K?8K is N?8K PFL 8I< J<<@E>gPFL ALJKJ<< 8 >I<<E :LG 4FL K?@EB `2?P CFFB K?@J :LG @J >I<<Ea EFK`/?<G@:KLI<@EDPM@JL8C:FIK<OF=K?@J:LG@J>I<<Ea

 E; @E K?< J8D< N8P N?<E G<FGC< 8I>L< FM<I N?<K?<I K?<

=8CC@E> KI<< D8B<J 8 JFLE; FI N?<K?<I +CLKF @J 8 GC8E<K K?<P ;FEcKJ<< K?<DJ<CM<J 8J 8I>L@E> FM<I N?<K?<I 8 :8K<>FI@Q8K@FE J?FLC; 9<8:K@M< @E K?<@I E<LI8C E<KNFIBJ $K J<<DJ C@B< <@K?<I K?< KI<< D8B<J8JFLE;FIEFK

2< BEFN N?<I< +CLKF @J 8E; N?<I< @KcJ >F@E> N< BEFN +CLKFcJJ?8G< 8E; +CLKFcJ D8JJg9LK @J @K 8 GC8E<K E; P<J K?<I< N<I<G<FGC< N?F J8@; K?@J N8J 8 =@>?K FM<I ;<=@E@K@FEJg9LK <M<E K?8K @J8 )<KNFIB JFIK F= G<IJG<:K@M< 9<:8LJ< PFLcI< 8I>L@E> 89FLK ?FN 

K?< :<EKI8C LE@K FL>?K KF 9< N@I<; LG $= PFL N<I< 8 D@E; :FEJKIL:Ke

-%&6+-!,'/'*+

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 227/309

<; 8CFE> K?< C@E<J F= )<KNFIB PFL NFLC;EcK J8P `$K ;<G<E;J FE?FN PFL ;<=@E< bGC8E<Kca PFL NFLC; ALJK J8P `"@M<E K?8K N< BEFN +CLKFcJ FI9@K 8E; J?8G< 8E; D8JJ K?<I< @J EF HL<JK@FE C<=K KF 8JBa

*I I8K?<I K?8KcJ ?FN @K NFLC; feel  g@K NFLC; feel  C@B< K?<I< N8J EFHL<JK@FE C<=Kg@= PFL N<I< 8 D@E; :FEJKIL:K<; 8CFE> K?< C@E<J F= )<KNFIB

<=FI< PFL :8E HL<JK@FE PFLI @EKL@K@FEJ PFL ?8M< KF I<8C@Q< K?8K N?8K PFLI D@E;cJ <P< @J CFFB@E> 8K is 8E @EKL@K@FEgJFD< :F>E@K@M<8C>FI@K?D 8J J<<E =IFD K?< @EJ@;<gI8K?<I K?8E 8 ;@I<:K G<I:<GK@FEF=K?<28P/?@E>J-<8CCPI<

+<FGC< :C@E> KF K?<@I @EKL@K@FEJ $ K?@EB EFK JF DL:? 9<:8LJ<K?<P 9<C@<M< K?<@I :F>E@K@M< 8C>FI@K?DJ 8I< G<I=<:KCP I<C@89C< 9LK9<:8LJ< K?<P :8EcK J<< K?<@I @EKL@K@FEJ as the way their cognitive algo- rithms happen to look from the inside.

 E; JF <M<IPK?@E> PFL KIP KF J8P 89FLK ?FN K?< E8K@M< :F>E@K@M<8C>FI@K?D >F<J 8JKI8P <E;J LG 9<@E> :FEKI8JK<; KF K?<@I ;@I<:K G<Ie:<GK@FE F= K?< 28P /?@E>J -<8CCP I<g8E; ;@J:8I;<; 8J F9M@FLJCP 

 NIFE>

http://lesswrong.com/lw/n1/allais_malaise/ 

'/ &$'*!,%$+*'%! &+!

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 228/309

11. Disputing Definitions

Followup to #FNEC>FI@K?D!<<CJ!IFD$EJ@;<

$ ?8M< N8K:?<; DFI< K?8E FE< :FEM<IJ8K@FEg<M<E :FEM<IJ8eK@FEJ JLGGFJ<;CP 89FLK :F>E@K@M< J:@<E:<g>F K?< IFLK< F= ;@JGLK@E> FM<I ;<=@E@K@FEJ /8B@E> K?< :C8JJ@: <O8DGC< KF 9< `$= 8 KI<< =8CCJ @E8 =FI<JK 8E; EF FE< ?<8IJ @K ;F<J @K D8B< 8 JFLE;a K?< ;@JGLK< F=eK<E=FCCFNJ8:FLIJ<C@B<K?@J

 If a tree falls in the forest, and no one hears it, does it make a sound? 

 C9<IK `*=:FLIJ<@K;F<J 2?8KB@E;F=J@CCPHL<JK@FE@JK?8K M<IPK@D<$cM<C@JK<E<;KF8KI<<=8CC@KD8;<8 JFLE;JF$cCC>L<JJK?8KFK?<IKI<<J=8CC@E>8CJFD8B<JFLE;J $;FEcK9<C@<M<K?<NFIC;:?8E><J8IFLE;N?<E$cDEFKCFFB@E>a

8IIP `28@K8D@ELK< $=EFFE<?<8IJ@K?FN:8E@K9<8JFLE;a

$E K?@J <O8DGC< 8IIP @J 8I>L@E> N@K? C9<IK 9<:8LJ< F= 8 ><EeL@E<CP ;@==<I<EK @EKL@K@FE 89FLK N?8K :FEJK@KLK<J 8 JFLE; LKK?<I<cJ DFI< K?8E FE< N8P K?< .K8E;8I; @JGLK< :8E JK8IK 8IIP :FLC; ?8M< 8 DFK@M< =FI I<A<:K@E> C9<IKcJ :FE:CLJ@FE *I 8IIP :FLC; 9< 8 JB<GK@: N?F LGFE ?<8I@E> C9<IKcJ 8I>LD<EK I<=C<O@M<CP J:ILK@E@Q<; @K =FI GFJJ@9C< CF>@:8C =C8NJ 8E; K?<E FE =@E;@E> 8 :FLEK<I8I>LD<EK 8LKFD8K@:8CCP 8::<GK<; @K N@K?FLK 8GGCP@E> 8 J<:FE; C8P<I F= J<8I:? =FI 8 :FLEK<Ie:FLEK<I8I>LD<EK K?<I<9P 8Ie

 >L@E> ?@DJ<C= @EKF K?< FGGFJ@K< GFJ@K@FE /?@J ;F<JEcK I<HL@I< K?8K8IIPcJ prior @EKL@K@FEgK?< @EKL@K@FE 8IIP NFLC; ?8M< ?8; @= N<c;8JB<;?@D9<=FI<C9<IKJGFB<g?8M<;@==<I<;=IFDC9<IKcJ

2<CC @= 8IIP ;@;EcK ?8M< 8 ;@==<I@E> @EKL@K@FE 9<=FI< ?< JLI<?8JFE<EFN

http://lesswrong.com/lw/np/disputing_definitions/  +8><b#FNEC>FI@K?D!<<CJ!IFD$EJ@;<c

+8><b&EFN@E>9FLK@8J<J8E#LIK+<FGC<c +8><b(FK@M8K<;.KFGG@E>8E;(FK@M8K<;FEK@EL8K@FEc

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 229/309

 C9<IK `2?8K;FPFLD<8EK?<I<cJEFJFLE; /?<KI<<cJIFFKJJE8GK?<KILEB:FD<J:I8J?@E>;FNE8E;?@KJK?<>IFLE;/?@J><E<I8K<JM@9I8K@FEJK?8KKI8M<C

K?IFL>?K?<>IFLE;8E;K?<8@I/?8KcJN?<I<K?<<E<I>P F=K?<=8CC>F<J@EKF?<8K8E;JFLE; I<PFLJ8P@E>K?8K@=G<FGC<C<8M<K?<=FI<JKK?<KI<<M@FC8K<J:FEJ<IM8K@FEF=<E<I>Pa

8IIP `LKEFFE< hears 8EPK?@E> $=K?<I<8I<EF?LD8EJ@EK?<=FI<JKFI=FIK?<J8B<F=8I>LD<EK8EPK?@E><CJ<N@K?8:FDGC<OE<IMFLJJPJK<D:8G89C<F= b?<8I@E>cK?<EEFFE<?<8IJ8JFLE;a

 C9<IK 8E; 8IIP I<:IL@K 8I>LD<EKJ K?8K feel  C@B< JLGGFIK =FIK?<@I I<JG<:K@M< GFJ@K@FEJ ;<J:I@9@E> @E DFI< ;<K8@C K?< K?FL>?KJK?8K :8LJ<; K?<@I `JFLE;ae;<K<:KFIJ KF =@I< FI JK8P J@C<EK LK JF =8IK?< :FEM<IJ8K@FE ?8J JK@CC =F:LJ<; FE K?< =FI<JK I8K?<I K?8E ;<=@E@eK@FEJ E; EFK< K?8K K?<P ;FEcK 8:KL8CCP ;@J8>I<< FE 8EPK?@E> K?8K?8GG<EJ@EK?<=FI<JK

 C9<IK `/?@J@JK?<;LD9<JK8I>LD<EK$cM<<M<I9<<E@E

4FLcI<8E@;;C<N@:B@E>=8CCLDG?@E>G@:BC<GCLD9<Ia

8IIP `4<8? 2<CCPFLCFFBC@B<PFLI=8:<:8L>?KFE=@I<8E;JFD<FE<GLK@KFLKN@K?8J?FM<Ca

$EJLCK ?8J 9<<E GIF==<I<; 8E; 8::<GK<; EFN E<@K?<I G8IKP :8E98:B ;FNE N@K?FLK CFJ@E> =8:< /<:?E@:8CCP K?@J @JEcK G8IK F= K?< argument, 8J I8K@FE8C@JKJ 8::FLEK JL:? K?@E>J 9LK @KcJ JL:? 8E @DGFIeK8EKG8IKF=K?<.K8E;8I;@JGLK<K?8K$cD@E:CL;@E>@K8EPN8P

 C9<IK `/?<KI<<GIF;L:<J8:FLJK@:M@9I8K@FEJ P ;<=@E@K@FEK?8K@J8JFLE;a

8IIP `)FFE<?<8IJ8EPK?@E> P;<=@E@K@FEK?8K@JEFK8JFLE;a

/?< 8I>LD<EK JK8IKJ J?@=K@E> KF =F:LJ FE ;<=@E@K@FEJ 2?<E<M<I PFL =<<C K<DGK<; KF J8P K?< NFI;J `9P ;<=@E@K@FEa @E 8E 8I>LD<EK

+8><b#FNEC>FI@K?D!<<CJ!IFD$EJ@;<c

!+(-,!&!&!,!'&+

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 230/309

K?8K @J EFK C@K<I8CCP 89FLK GLI< D8K?<D8K@:J I<D<D9<I K?8K 8EPeK?@E> N?@:? @J KIL< `9P ;<=@E@K@FEa @J KIL< @E 8CC GFJJ@9C< NFIC;J 8E;JFF9J<IM@E>@KJKILK?:8EE<M<I :FEJKI8@E which NFIC;PFLC@M<@E

 C9<IK`(P:FDGLK<IcJD@:IFG?FE<:8EI<:FI;8JFLE; N@K?FLK8EPFE<9<@E>8IFLE;KF?<8I@KJKFI<@K8J8=@C<8E;@KcJ:8CC<;8bJFLE;=@C<cE;N?8KcJJKFI<;@EK?<=@C<@JK?<G8KK<IEF=M@9I8K@FEJ@E8@IEFKK?<G8KK<IEF= E<LI8C=@I@E>J@E8EPFE<cJ9I8@E b.FLE;cD<8EJ8G8KK<IEF=M@9I8K@FEJa

 C9<IK ;<GCFPJ 8E 8I>LD<EK K?8K feels C@B< JLGGFIK =FI K?< NFI;

`JFLE;a having a particular meaning. /?@J @J 8 ;@==<I<EK B@E; F= HL<JeK@FE =IFD N?<K?<I 8:FLJK@: M@9I8K@FEJ K8B< GC8:< @E 8 =FI<JKg9LKK?<J?@=KLJL8CCPG8JJ<JLEEFK@:<;

8IIP `*?P<8? '<KcJALJKJ<<@=K?<;@:K@FE8IP8>I<<J N@K?PFLa

/?<I<cJ 8 CFK F= K?@E>J $ :FLC; 9< :LI@FLJ 89FLK @E K?< =8CC@E>eKI<< J:<E8I@F $ :FLC; >F @EKF K?< =FI<JK 8E; CFFB 8K KI<<J FI C<8IE

?FN KF ;<I@M< K?< N8M< <HL8K@FE =FI :?8E><J F= 8@I GI<JJLI< FI <Oe8D@E< K?< 8E8KFDP F= 8E <8I FI JKL;P K?< E<LIF8E8KFDP F= K?<8L;@KFIP :FIK<O $EJK<8; F= ;F@E> 8EP F= K?<J< K?@E>J $ 8D KF:FEJLCK 8 ;@:K@FE8IP 8GG8I<EKCP 2?P I< K?< <;@KFIJ F= K?< ;@:eK@FE8IP <OG<IK 9FK8E@JKJ <OG<IK G?PJ@:@JKJ <OG<IK E<LIFJ:@<EK@JKJ'FFB@E> @E 8E <E:P:CFG<;@8 D@>?K D8B< J<EJ< 9LK N?P 8 dictionary? 

 C9<IK `#8? <=@E@K@FE:@E(<II@8De2<9JK<Ib.FLE; (<:?8E@:8CI8;@8EK<E<I>PK?8K@JKI8EJD@KK<;9P 

CFE>@KL;@E8CGI<JJLI<N8M<J@E8D8K<I@8CD<;@LDh8J8@Iica

8IIP `#8? <=@E@K@FE9@E(<II@8De2<9JK<Ib.FLE; /?<J<EJ8K@FEG<I:<@M<;9PK?<J<EJ<F= ?<8I@E>ca

+8><b/?<+8I89C<F=#<DCF:Bc +8><b!8B<OGC8E8K@FEJc

-%&6+-!,'/'*+

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 231/309

 C9<IK8E;8IIP:?FILJ `FEJ8IE<;;@:K@FE8IP /?@J;F<JEcK?<CG8K8CCa

@:K@FE8IP <;@KFIJ 8I< ?@JKFI@8EJ F= LJ8>< EFK C<>@JC8KFIJ F= C8Ee >L8>< @:K@FE8IP <;@KFIJ =@E; NFI;J @E :LII<EK LJ8>< K?<E NI@K<;FNE K?< NFI;J E<OK KF h8 JD8CC G8IK F=  i N?8K G<FGC< J<<D KF D<8E9P K?<D $= K?<I<cJ DFI< K?8E FE< LJ8>< K?< <;@KFIJ NI@K< ;FNEDFI<K?8EFE<;<=@E@K@FE

 C9<IK `'FFBJLGGFJ<K?8K$C<=K8D@:IFG?FE<@EK?<=FI<JK8E;I<:FI;<;K?<G8KK<IEF=K?<8:FLJK@:M@9I8K@FEJF=K?<KI<<=8CC@E> $=$GC8P<;K?8K98:BKFJFD<FE<

K?<Pc;:8CC@K8bJFLE;c /?8KcJK?<:FDDFELJ8>< FEcK >F8IFLE;D8B@E>LGPFLIFNEN8:BP;<=@E@K@FEJa

8IIP `*E<$:8E;<=@E<8NFI;8EPN8P$C@B<JFCFE> 8J$LJ<@K:FEJ@JK<EKCP /NFK?<D<8E@E>$>8M<N8J inK?<;@:K@FE8IP /?I<<N?F>8M< you K?<I@>?KKF;<:@;<

 N?8K@JFI@JEcK:FDDFELJ8><a

/?<I<cJ HL@K< 8 CFK F= I8K@FE8C@KP <IIFIJ @E K?< .K8E;8I; @JGLK<

.FD< F= K?<D $cM< 8CI<8;P :FM<I<; 8E; JFD< F= K?<D $cM< P<K KF:FM<IC@B<N@J<K?<I<D<;@<J

LK =FI EFN $ NFLC; ALJK C@B< KF GF@EK FLKg@E 8 DFLIE=LC JFIKF= N8PgK?8K C9<IK 8E; 8IIP J<<D KF 8>I<< FE M@IKL8CCP <M<IP HL<JeK@FE F= N?8K @J actually >F@E> FE @EJ@;< K?< =FI<JK 8E; P<K @K ;F<JEcKJ<<DKF><E<I8K<8EP=<<C@E>F=8>I<<D<EK

 I>L@E> 89FLK ;<=@E@K@FEJ @J 8 >8I;<E G8K? G<FGC< NFLC;EcK >F;FNE K?< G8K? @= K?<P J8N 8K K?< FLKJ<K N?<I< @K C<; $= PFL 8JB<;

 C9<IK h8IIPi N?P ?<cJ JK@CC 8I>L@E> ?<c; GIF989CP J8P JFD<K?@E> C@B< `8IIP hC9<IKi @J KIP@E> KF JE<8B @E ?@J FNE ;<=@E@K@FE F= bJFLE;c K?< J:LIM<P J:FLE;I<C KF JLGGFIK ?@J I@;@:LCFLJ GF@EK 8E;$cD?<I<KF;<=<E;K?<JK8E;8I;;<=@E@K@FEa

LK JLGGFJ< $ N<EK 98:B @E K@D< KF 9<=FI< K?< JK8IK F= K?< 8I>LeD<EK

+8><bOK<EJ@FEJ8E;$EK<EJ@FEJc

!+(-,!&!&!,!'&+

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 232/309

(Eliezer appears from nowhere in a peculiar conveyance that  looks just like the time machine from the original ‘The Time Machine’ movie.)

8IIP `"FJ? K@D<KI8M<C<Ia

C@<Q<I `$8D8KI8M<C<I=IFDK?<=LKLI< #<8IDP  NFI;J $?8M<KI8M<C<;=8I@EKFK?<G8JKg8IFLE;=@=K<<ED@ELK<Jga

 C9<IK `!@=K<<E minutes? a

C@<Q<I `gKF9I@E>PFLK?@JD<JJ8><a

(There is a pause of mixed confusion and expectancy.)

C@<Q<I `FPFLK?@EBK?8KbJFLE;cJ?FLC;9<;<=@E<;KFI<HL@I<9FK?8:FLJK@:M@9I8K@FEJhGI<JJLI<N8M<J@E8@Ii8E;8CJF8L;@KFIP<OG<I@<E:<JhJFD<FE<KFC@JK<EKFK?<JFLE;iFIJ?FLC;bJFLE;c9<;<=@E<;8JD<8E@E>FECP 8:FLJK@:M@9I8K@FEJFIFECP8L;@KFIP<OG<I@<E:<a

8IIP `4FLN<EK98:B@EK@D<KF8JBLJ that? a

C@<Q<I `(PGLIGFJ<J8I<DPFNE EJN<Ia

 C9<IK `2<CC_$;FEcKJ<<N?P@KNFLC;D8KK<I 4FL:8EG@:B8EP;<=@E@K@FEJFCFE>8JPFLLJ<@K:FEJ@JK<EKCPa

8IIP `!C@G8:F@E I=C@G8:F@EKN@:<a

C@<Q<I `+<IJFE8CCP$c;J8PK?8K@=K?<@JJL<8I@J<J9FK?J@;<JJ?FLC;JN@K:?KF;<J:I@9@E>K?<<M<EK@ELE8D9@>LFLJCFN<IeC<M<C:FEJK@KL<EKJC@B<8:FLJK@:

 M@9I8K@FEJFI8L;@KFIP<OG<I@<E:<J *I<8:?J@;<:FLC;;<J@>E8K<8E<NNFI;C@B<b8C9<IQC<c8E;b98I>LCLDcKFLJ<=FIN?8KK?<PI<JG<:K@M<CPLJ<;KF:8CCbJFLE;c8E;K?<E9FK?J@;<J:FLC;LJ<K?<E<NNFI;J:FEJ@JK<EKCP/?8KN8PE<@K?<IJ@;<?8JKF98:B;FNEFICFJ<=8:<9LK

K?<P:8EJK@CC:FDDLE@:8K< E;F=:FLIJ<PFLJ?FLC;

-%&6+-!,'/'*+

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 233/309

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 234/309

12. Feel the Meaning

Followup to @JGLK@E><=@E@K@FEJ

2?<E $ ?<8I JFD<FE< J8P `*? CFFB 8 9LKK<I=CPa K?< JGFB<EG?FE<D<J `9LKK<I=CPa <EK<I DP <8I 8E; M@9I8K< FE DP <8I ;ILD 9<e@E> KI8EJD@KK<; KF K?< :F:?C<8 K@:BC@E> 8L;@KFIP E<IM<J K?8K KI8EJeD@K 8:K@M8K@FE JG@B<J KF K?< 8L;@KFIP :FIK<O N?<I< G?FE<D< GIFe:<JJ@E> 9<>@EJ 8CFE> N@K? I<:F>E@K@FE F= NFI;J 8E; I<:FEJKIL:K@FEF= JPEK8O h8 9P EF D<8EJ J<I@8C GIF:<JJi 8E; 8CC D8EE<I F= FK?<I:FDGC@:8K@FEJ

LK 8K K?< <E; F= K?< ;8P FI I8K?<I 8K K?< <E; F= K?< J<:FE;

$ 8D GI@D<; KF CFFB N?<I< DP =I@<E; @J GF@EK@E> 8E; J<< 8 M@JL8CG8KK<IE K?8K $ N@CC I<:F>E@Q< 8J 8 9LKK<I=CP 8E; $ NFLC; 9< HL@K< JLIeGI@J<;KFJ<<8NFC=@EJK<8;

(P =I@<E; CFFBJ 8K 8 9LKK<I=CP ?@J K?IF8K M@9I8K<J 8E; C@GJ DFM<K?< GI<JJLI< N8M<J KI8M<C @EM@J@9CP K?IFL>? K?< 8@I DP <8I ?<8IJ8E; DP E<IM<J KI8EJ;L:< 8E; DP 9I8@E I<:FEJKIL:KJ 8E; CF 8E; 9<e?FC; $ BEFN N?8K DP =I@<E; @J CFFB@E> 8K $JEcK K?8K D8IM<CFLJ $= 

 N< ;@;EcK BEFN 89FLK K?< GI<JJLI< N8M<J @E K?< 8@I @K NFLC; 9< 8 KI<D<E;FLJ ;@J:FM<IP @E 8CC K?< E<NJG8G<IJ #LD8EJ 8I< K<C<G8K?e

@: #LD8E9I8@EJ:8EKI8EJ=<IK?FL>?KJKF<8:?FK?<I2<CC N< are K<C<G8K?@: @E =8:K 9LK D8>@: @JEcK <O:@K@E> N?<E @KcJ

D<I<CP real 8E;8CCPFLI=I@<E;J:8E;F@KKFF

/?@EB K<C<G8K?P @J J@DGC< /IP 9L@C;@E> 8 :FDGLK<I K?8K N@CC9< K<C<G8K?@: N@K? PFL /<C<G8K?P FI `C8E>L8><a FI N?8K<M<I PFL

 N8EK KF :8CC FLI G8IK@8C K?FL>?K KI8EJ=<I 89@C@KP @J DFI< :FDGC@:8K<;K?8E@KCFFBJ

LK @K NFLC; 9< HL@K< @E:FEM<E@<EK KF >F 8IFLE; K?@EB@E> `)FN 

$ J?8CC G8IK@8CCP KI8EJ;L:< JFD< =<8KLI<J F= DP K?FL>?KJ @EKF 8 C@E<8IJ<HL<E:< F= G?FE<D<J N?@:? N@CC @EMFB< J@D@C8I K?FL>?KJ @E DP :FEM<IJ8K@FE8CG8IKE<I_a

.F K?< 9I8@E ?@;<J K?< :FDGC<O@KPgFI I8K?<I E<M<I I<GI<J<EKJ@K @E K?< =@IJK GC8:<gN?@:? C<8;J G<FGC< KF K?@EB JFD< G<:LC@8IK?FL>?KJ89FLKNFI;J

http://lesswrong.com/lw/nq/feel_the_meaning/ 

+8><b@JGLK@E><=@E@K@FEJc +8><b`.:@<E:<a8JLI@[email protected]<Ic

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 235/309

 J $ I<D8IB<; <8IC@<I N?<E 8 C8I>< P<CCFN JKI@G<; F9A<:K C<8GJ8K D< $ K?@EB `4@B<J K@><Ia EFK `#D_ F9A<:KJ N@K? K?< GIFG<IeK@<J F= C8I><E<JJ P<CCFNE<JJ 8E; JKI@G<;E<JJ ?8M< GI<M@FLJCP F=K<E

GFJJ<JJ<; K?< GIFG<IK@<J b?LE>IPc 8E; b;8E><IFLJc 8E; K?<I<=FI< 8CeK?FL>? @K @J EFK CF>@:8CCP E<:<JJ8IP auughhhh -0)# -0)#"0'+a

.@D@C8ICP N?<E JFD<FE< J?FLKJ `4@B<J K@><Ia E8KLI8C J<C<:eK@FE NFLC; EFK =8MFI 8E FI>8E@JD K?8K K?FL>?K `#D_ $ ?8M< ALJK?<8I; K?< JPCC89C<J b/@<c 8E; b"IIc N?@:? DP =<CCFN KI@9< D<D9<IJ8JJF:@8K< N@K? K?<@I @EK<IE8C 8E8CF>L<J F= DP FNE tiger :FE:<GK 8E;

 N?@:? K?<P 8I< DFI< C@B<CP KF LKK<I @= K?<P J<< 8E F9A<:K K?<P :8K<>FeI@Q< 8J aiiieeee -0)# -0)# help it’s got my arm -0)#

"0'+aC<>>7FEe

J@;<I@E> K?@J 8J 8 ;<J@>E:FEeJKI8@EK FEK?< ?Le

D8E:F>E@K@M<8I:?@K<:eKLI< PFL

 NFLC;EcK N8EK any<OKI8 JK<GJ 9<eKN<<E

 N?<E PFLI 8Le;@KFIP :FIK<O I<:F>eE@Q<J K?< JPCC89C<J `K@><Ia 8E; N?<E K?< K@><I :FE:<GK ><KJ8:K@M8K<;

+8><b2FI;J8J#@;;<E$E=<I<E:<Jc +8><b#FNEC>FI@K?D!<<CJ!IFD$EJ@;<c

$,%&!&

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 236/309

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 237/309

8IIP `*?P<8? '<KcJALJKJ<<@=K?<;@:K@FE8IP8>I<<J N@K?PFLa

 C9<IK =<<CJ @EKL@K@M<CP K?8K K?< NFI; `JFLE;a has a meaning 8E;K?8K K?< D<8E@E>  is 8:FLJK@: M@9I8K@FEJ %LJK 8J C9<IK =<<CJ K?8K 8 KI<< =8CC@E> @E K?< =FI<JK makes a sound hI8K?<I K?8E :8LJ@E> 8E <M<EKK?8K matches the sound category i

8IIPC@B<N@J< feels K?8K

GCIB9A:6B>B<6I9>HCFM:LD:F>:B8:G

;CF:GHGCIB9;6@G:

-8K?<IK?8E

AMF6>B!>B9CB8:DHVGCIB9W

8CB8:DH5I9>HCFMLD:F>:B8:

8CB8:DH5I9>HCFMLD:F>:B8:A6H8=;CF:GH

;6@G:

2?@:? @J :CFJ<I KF N?8KcJ really >F@E> FE 9LK ?LD8EJ ?8M< EFK<MFCM<; KF BEFN K?@J 8EPDFI< K?8E ?LD8EJ @EJK@E:K@M<CP BEFN K?<

9I8@E@JD8;<F=E<LIFEJ C9<IK 8E; 8IIPcJ :FE=C@:K@E> @EKL@K@FEJ GIFM@;< K?< =L<C =FI

:FEK@EL@E> K?< 8I>LD<EK @E K?< G?8J< F= 8I>L@E> FM<I N?8K K?< NFI;`JFLE;a D<8EJgN?@:? feels C@B< 8I>L@E> FM<I 8 =8:K C@B< 8EP FK?<I=8:KC@B<8I>L@E>FM<IN?<K?<IK?<JBP@J9CL<FI>I<<E

4FL D8P EFK <M<E EFK@:< K?8K 8EPK?@E> ?8J >FE< 8JKI8P LEK@C PFL KIP KF G<I=FID K?< I8K@FE8C@JK I@KL8C F= JK8K@E> 8 K<JK89C< <OeG<I@D<EK  N?FJ< I<JLCK ;<G<E;J FE K?< =8:KJ PFLcI< JF ?<8K<;CP 

;@JGLK@E>_

+8><b(8B@E><C@<=J+8P-<EKh@EEK@:@G8K<;OG<I@<E:<Jic

$,%&!&

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 238/309

13. The Argument from Common Usage

Followup to !<<CK?<(<8E@E> 

+8IKF=K?< .K8E;8I;<=@E@K@FE8C@JGLK< ILEJ8J=FCCFNJ

 C9<IK `'FFBJLGGFJ<K?8K$C<=K8D@:IFG?FE<@EK?<=FI<JK8E;I<:FI;<;K?<G8KK<IEF=K?<8:FLJK@:M@9I8K@FEJF=K?<KI<<=8CC@E> $=$GC8P<;K?8K98:BKFJFD<FE<K?<Pc;:8CC@K8bJFLE;c /?8KcJK?<:FDDFELJ8>< FEcK

 >F8IFLE;D8B@E>LGPFLIFNEN8:BP;<=@E@K@FEJa

8IIP `*E<$:8E;<=@E<8NFI;8EPN8P$C@B<JFCFE> 8J$LJ<@K:FEJ@JK<EKCP /NFK?<D<8E@E>$>8M<N8J inK?<;@:K@FE8IP /?I<<N?F>8M< you K?<I@>?KKF;<:@;<

 N?8K@JFI@JEcK:FDDFELJ8><a

)FK 8CC ;<=@E@K@FE8C ;@JGLK<J GIF>I<JJ 8J =8I 8J I<:F>E@Q@E> K?<EFK@FE F= :FDDFE LJ8>< (FI< F=K<E $ K?@EB JFD<FE< G@:BJLG 8 ;@:K@FE8IP 9<:8LJ< K?<P 9<C@<M< K?8K NFI;J ?8M< D<8E@E>J8E; K?< ;@:K@FE8IP =8@K?=LCCP I<:FI;J N?8K K?@J D<8E@E> @J .FD<

G<FGC< <M<E J<<D KF 9<C@<M< K?8K K?< ;@:K@FE8IP determines K?< D<8Ee@E>gK?8K K?< ;@:K@FE8IP <;@KFIJ 8I< K?< '<>@JC8KFIJ F= '8E>L8><(8P9< 9<:8LJ< 98:B @E <C<D<EK8IP J:?FFC K?<@I 8LK?FI@KPeK<8:?<I

J8@; K?8K K?<P ?8; KF F9<P K?< ;@:K@FE8IP K?8K @K N8J 8 D8E;8KFIP ILC<I8K?<IK?8E8EFGK@FE8CFE<

@:K@FE8IP <;@KFIJ I<8; N?8K FK?<I G<FGC< NI@K< 8E; I<:FI; N?8K K?< NFI;J J<<D KF D<8E K?<P 8I< ?@JKFI@8EJ /?< *O=FI; Ee >C@J?@:K@FE8IPD8P9< comprehensive, 9LKE<M<I authoritative.

LK JLI<CP K?<I< @J 8 JF:@8C @DG<I8K@M< KF LJ< NFI;J @E 8 :FDeDFECP LE;<IJKFF; N8P F<J EFK FLI ?LD8E K<C<G8K?P FLI M8CLe89C< GFN<I F= C8E>L8>< I<CP FE DLKL8C :FFI;@E8K@FE KF NFIB +<Ie?8GJ N< J?FLC; MFCLEK8I@CP KI<8K ;@:K@FE8IP <;@KFIJ 8J JLGI<D< 8Ie9@K<IJg<M<E @=  they GI<=<I KF K?@EB F= K?<DJ<CM<J 8J ?@JKFI@8EJg@E

http://lesswrong.com/lw/nr/the_argument_from_common_usage/  +8><b!<<CK?<(<8E@E>c +8><b@JGLK@E><=@E@K@FEJc

+8><b!<<CK?<(<8E@E>c +8><b9JFCLK<LK?FI@KPc

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 239/309

FI;<I KF D8@EK8@E K?< HL@<K :FFG<I8K@FE FE N?@:? 8CC JG<<:? ;<eG<E;J

/?< G?I8J< `8LK?FI@K8K@M< ;@:K@FE8IPa @J 8CDFJK E<M<I LJ<; :FIe

I<:KCP 8E <O8DGC< F= GIFG<I LJ8>< 9<@E> K?< LK?FI@K8K@M< @:K@FeE8IP F= $ .K8E;8I;J /?< $ @J 8 9F;P F= MFK@E> D<D9<IJ

 N?F ?8M< 8 GIF=<JJ@FE8C E<<; =FI <O8:K 8>I<<D<EK FE K<IDJ 8E; ;<=e@E@K@FEJ 8E; JF K?< LK?FI@K8K@M< @:K@FE8IP F= $ .K8E;8I;J @J8:KL8C E<>FK@8K<; C<>@JC8K@FE N?@:? <O<IKJ N?8K<M<I 8LK?FI@KP FE<I<>8I;J8JI<J@;@E>@EK?<$

$E <M<IP;8P C@=< J?8I<; C8E>L8>< LJL8CCP ;F<J EFK 8I@J< =IFD 8 ;<C@9<I8K< 8>I<<D<EK 8J F= K?< $ $KcJ DFI< 8 D8KK<I F= @Ee=<:K@FE 8J NFI;J 8I< @EM<EK<; 8E; ;@==LJ< K?IFL>? K?< :LCKLI< h

`D<D<a FE< D@>?K J8P =FCCFN@E> -@:?8I; 8NB@EJ K?@IKP P<8IJ8>Fg9LK PFL 8CI<8;P BEFN N?8K $ D<8E 8E; @= EFK PFL :8E CFFB @KLGFE"FF>C<8E;K?<EPFLKFFN@CC?8M<9<<E@E=<:K<;i

4<K 8J K?< <O8DGC< F= K?< $ J?FNJ 8>I<<D<EK FE C8E>L8><:8E 8CJF 9< 8 :FFG<I8K@M<CP <JK89C@J?<; GL9C@: >FF; $= PFL 8E; $

 N@J? KF LE;<I>F 8E <O:?8E>< F= K?FL>?KJ M@8 C8E>L8>< K?< ?LD8EK<C<G8K?P K?<E @K @J @E FLI DLKL8C @EK<I<JK K?8K N< LJ< K?< same NFI;=FI J@D@C8I :FE:<GKJgGI<=<I89CP :FE:<GKJ J@D@C8I KF K?< C@D@K F= 

I<JFCLK@FE @E FLI 9I8@EcJ I<GI<J<EK8K@FE K?<I<F=g<M<E K?FL>? N<?8M< EF F9M@FLJ DLKL8C @EK<I<JK @E LJ@E> 8EP  particular NFI; =FI 8 :FE:<GK

2< ?8M< EF F9M@FLJ DLKL8C @EK<I<JK @E LJ@E> K?< NFI; `FKFa KFD<8E JFLE; FI `JFLE;a KF D<8E FKF 9LK N< ?8M< 8 DLKL8C @EK<I<JK@E LJ@E> K?< same NFI; N?@:?<M<I NFI; @K ?8GG<EJ KF 9< h+I<=<Ie89CP NFI;J N< LJ< =I<HL<EKCP J?FLC; 9< J?FIK 9LK C<KcJ EFK ><K @EKF@E=FID8K@FEK?<FIPALJKP<Ki

LK N?@C< N< ?8M< 8 DLKL8C @EK<I<JK @K @J EFK JKI@:KCP  necessary

K?8K PFL 8E; $ LJ< K?< J@D@C8I C89<CJ internally @K @J FECP :FEM<E@<EK$= $ BEFN K?8K KF PFL `FKFa D<8EJ JFLE;gK?8K @J PFL 8JJF:@8K<`FKFa KF 8 :FE:<GK M<IP J@D@C8I KF K?< FE< $ 8JJF:@8K< KF`JFLE;agK?<E $ :8E J8P `+8G<I :ILDGC@E> D8B<J 8 :I8:BC@E> FKFa$KI<HL@I<J<OKI8K?FL>?K9LK$:8E;F@K@=$N8EK

.@D@C8ICP @= PFL J8P `2?8K @J K?< N8CB@E>eJK@:B F= 8 9FNC@E> 98CC;IFGG@E> FE K?< =CFFIa 8E; $ BEFN N?@:? :FE:<GK you 8JJF:@8K<

 N@K? K?< JPCC89C<J `N8CB@E>eJK@:Ba K?<E $ :8E =@>LI< FLK N?8K PFL

D<8E $K D8P I<HL@I< JFD< K?FL>?K 8E; >@M< D< G8LJ< 9<:8LJ< $

,*-%&,*'%'%%'&-+

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 240/309

FI;@E8I@CP 8JJF:@8K< `N8CB@E>eJK@:Ba N@K? 8 ;@==<I<EK :FE:<GK LK $:8E;F@KALJK=@E<

2?<E ?LD8EJ I<8CCP  want  KF :FDDLE@:8K< N@K? <8:? FK?<I

 N<cI< ?8I; KF JKFG $= N<cI< JKL:B FE 8 ;<J<IK<; @JC8E; N@K? EF :FDeDFEC8E>L8><N<cCCK8B<LGJK@:BJ8E;;I8NG@:KLI<J@EJ8E;

 C9<IKcJ 8GG<8C KF K?< I>LD<EK =IFD FDDFE 0J8>< 8JJLD<JK?8K 8>I<<D<EK FE C8E>L8>< @J 8 :FFG<I8K@M<CP <JK89C@J?<; GL9C@:

 >FF; 4<K C9<IK 8JJLD<J K?@J =FI K?< JFC< GLIGFJ< F= I?<KFI@:8CCP 8::LJ@E> 8IIP F= 9I<8B@E> K?< 8>I<<D<EK 8E; <E;8E><I@E> K?<GL9C@: >FF; )FN K?< =8CC@E>eKI<< 8I>LD<EK ?8J >FE< 8CC K?< N8P =IFD 9FK8EP KF J<D8EK@:J KF GFC@K@:J 8E; JF 8IIP I<JGFE;J 9P :?8CeC<E>@E>C9<IK=FIK?<8LK?FI@KPKF;<=@E<K?<NFI;

  I8K@FE8C@JK N@K? K?< ;@J:@GC@E< F= ?L>>@E> K?< HL<IP  8:K@M< NFLC;EFK@:<K?8KK?<:FEM<IJ8K@FE?8;>FE<I8K?<I=8I8JKI8P

*? ;<8I I<8;<I @J @K 8CC I<8CCP E<:<JJ8IP C9<IK BEFNJ N?8K8IIP D<8EJ 9P `JFLE;a 8IIP BEFNJ N?8K C9<IK D<8EJ 9P `JFLE;a FK? C9<IK 8E; 8IIP ?8M< 8::<JJ KF NFI;J JL:? 8J`8:FLJK@: M@9I8K@FEJa FI `8L;@KFIP <OG<I@<E:<a N?@:? K?<P 8CI<8;P 8JJF:@8K< KF K?< J8D< :FE:<GKJ 8E; N?@:? :8E ;<J:I@9< <M<EKJ @EK?< =FI<JK N@K?FLK 8D9@>L@KP $= K?<P N<I< JKL:B FE 8 ;<J<IK<; @Je

C8E; KIP@E> KF :FDDLE@:8K< N@K? <8:? FK?<I K?<@I NFIB NFLC; 9< done.

2?<E 9FK? J@;<J know N?8K K?< FK?<I J@;< wants KF J8P 8E;9FK? J@;<J 8::LJ< K?< FK?<I J@;< F= ;<=<:K@E> =IFD `:FDDFE LJ8><aK?<E N?8K<M<I @K @J K?<P 8I< 89FLK @K @J :C<8ICP EFK working out away to communicate with each other. LK K?@J @J K?< N?FC< 9<E<=@K K?8K:FDDFELJ8><GIFM@;<J@EK?<=@IJKGC8:<

2?P NFLC; PFL 8I>L< 89FLK K?< D<8E@E> F= 8 NFI; KNF J@;<JKIP@E> KF NI<JK @K 98:B 8E; =FIK? $= @KcJ ALJK 8 E8D<JG8:< :FE=C@:KK?8K ?8J >FKK<E 9CFNE FLK F= GIFGFIK@FE 8E; EFK?@E> DFI< @J 8KJK8B< K?<E K?< KNF J@;<J E<<; D<I<CP ><E<I8K< KNF E<N NFI;J 8E;LJ<K?<D:FEJ@JK<EKCP 

4<K F=K<E :8K<>FI@Q8K@FEJ =LE:K@FE 8J ?@;;<E @E=<I<E:<J 8E;;@J>L@J<; HL<I@<J $J 8K?<@JD 8 `I<C@>@FEa $= JFD<FE< @J 8I>L@E> 

+8><b#L>K?<,L<IPc +8><b@JGLK@E><=@E@K@FEJc +8><b2FI;J8J#@;;<E$E=<I<E:<Jc

+8><b@J>L@J<;,L<I@<Jc +8><b0E:I@K@:8C.LG<I:I@K@:8C@KPc

-%&6+-!,'/'*+

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 241/309

K?8K K?< I<8JFE@E> D<K?F;J LJ<; @E 8K?<@JD 8I< FE 8 G8I N@K? K?<I<8JFE@E> D<K?F;J LJ<; @E %L;8@JD FI K?8K 8K?<@JD @J FE 8 G8I N@K?$JC8D @E K<IDJ F= :8LJ8CCP <E><E;<I@E> M@FC<E:< K?<E K?<P ?8M< 8 

:C<8I 8I>LD<EK8K@M< JK8B< @E CLDG@E> @K 8CC KF><K?<I @EKF 8E @E;@JeK@E:K>I8P9CLIF=`=8@K?`

*I :FEJ@;<I K?< =@>?K KF 9C<E; KF><K?<I 9C8:BJ 8E; N?@K<J 8J`G<FGC<a /?@J NFLC; EFK 9< 8 K@D< KF ><E<I8K< KNF NFI;JgN?8KcJ8K JK8B< @J <O8:KCP K?< @;<8 K?8K PFL J?FLC;EcK ;I8N 8 DFI8C ;@JK@E:eK@FE

LK FE:< 8EP <DG@I@:8C GIFGFJ@K@FE @J 8K JK8B< or  8EP DFI8CGIFGFJ@K@FEPFL:8EEFCFE><I8GG<8CKF:FDDFELJ8><

$= K?< HL<JK@FE @J ?FN KF :CLJK<I KF><K?<I J@D@C8I K?@E>J

=FIGLIGFJ<J F= @E=<I<E:< <DG@I@:8C GI<;@:K@FEJ N@CC ;<G<E; FE K?< 8EeJN<I N?@:? D<8EJ K?8K ;<=@E@K@FEJ :8E 9< wrong  :FE=C@:K F= GI<;@:K@FEJ:8EEFK9<J<KKC<;9P8EFG@E@FEGFCC

$= PFL N8EK KF BEFN N?<K?<I 8K?<@JD J?FLC; 9< :CLJK<I<; N@K?JLG<IE8KLI8C@JK I<C@>@FEJ =FI GLIGFJ<J F= JFD< G8IK@:LC8I <DG@I@:8C@E=<I<E:<K?<;@:K@FE8IP:8EcK8EJN<IPFL

$= PFL N8EK KF BEFN N?<K?<I 9C8:BJ 8I< G<FGC< K?< ;@:K@FE8IP :8EcK8EJN<IPFL

$= <M<IPFE< 9<C@<M<J K?8K K?< I<; C@>?K @E K?< JBP @J (8IJ K?< "F;F= 28I K?< ;@:K@FE8IP N@CC ;<=@E< `(8IJa 8J K?< "F; F= 28I $= <M<IPFE< 9<C@<M<J K?8K =@I< @J K?< I<C<8J< F= G?CF>@JKFE K?< ;@:K@FeE8IPN@CC;<=@E<`=@I<a8JK?<I<C<8J<F=G?CF>@JKFE

/?<I< @J 8E 8IK KF LJ@E> NFI;J <M<E N?<E ;<=@E@K@FEJ 8I< EFKC@K<I8CCP KIL< FI =8CJ< K?<P 8I< F=K<E N@J<I FI DFI< =FFC@J? @:K@FeE8I@<J 8I< D<I< ?@JKFI@<J F= G8JK LJ8>< @= PFL KI<8K K?<D 8J JLGI<D<8I9@K<IJ F= D<8E@E> @K 9@E;J PFL KF K?< N@J;FD F= K?< G8JK =FIe

9@;;@E>PFLKF;F9<KK<I/?FL>? ;F K8B< :8I< KF <EJLI< h@= PFL DLJK ;<G8IK =IFD K?< N@Je

;FD F= K?< G8JKi K?8K G<FGC< :8E =@>LI< FLK N?8K PFLcI< KIP@E> KFJN@D

+8><b/?<!8CC8:PF="I8Pc +8><b.@D@C8I@KPCLJK<IJc +8><bOK<EJ@FEJ8E;$EK<EJ@FEJc

+8><b"L8I;@8EJF=K?</ILK?c http://lesswrong.com/lw/h9/tsuyoku_vs_the_egalitarian_instinct/ 

,*-%&,*'%'%%'&-+

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 242/309

14. Empty Labels

Followup to /?<I>LD<EK=IFDFDDFE0J8><

FEJ@;<I hP<K 8>8@Ei K?< I@JKFK<C@8E @;<8 F= :8K<>FI@<J '<KcJJ8P K?8K K?<I<cJ JFD< F9A<:K N@K? GIFG<IK@<J 8E; FI 8KC<8JK@KCFFBJe@J?

!I<; `4FLD<8EK?8KK?@E>FM<IK?<I<@J9CL<IFLE;=LQQP8E;ga(<`$EI@JKFK<C@8ECF>@:@KcJEFKJLGGFJ<;KFD8B<8 ;@==<I<E:<N?8KK?<GIFG<IK@<J8I<FIN?8K$:8CCK?<D/?8KcJN?P$cDALJKLJ@E>K?<C<KK<IJa

)<OK $ @EM<EK K?< I@JKFK<C@8E :8K<>FIP `Q8N8a N?@:? ;<J:I@9<JK?FJ< F9A<:KJ 8CC K?FJ< F9A<:KJ 8E; FECP K?FJ< F9A<:KJ N?@:? ?8M<GIFG<IK@<J8E;

(< `*9A<:K@JQ8N88E;a!I<; `E;@KcJ9CL<g$D<8EgKFFI@>?Ka(< `/?8KcJ@DGC@<;N?<E$J8P@KcJQ8N8a

!I<; `.K@CC$c;C@B<PFLKFJ8P@K<OGC@:@KCPa(< `*B8P *9A<:K@JQ8N88E;a

/?<E $ 8;; 8EFK?<I NFI; `PFB@<a N?@:? ;<J:I@9<J 8CC 8E; FECP F9A<:KJ K?8K 8I< 8E; 8E; K?< NFI; `O@GGFa N?@:? ;<J:I@9<J 8CC8E;FECPF9A<:KJN?@:?8I<9LKEFK

(< `*9A<:K@JQ8N88E;PFB@<9LKEFKO@GGFa!I<; `28@K@J@KCLD@E<J:<EK $D<8E@J@Ka

(< `4<J /?8K@JK?<FECPGFJJ@9@C@KPFEK?<@E=FID8K@FE>@M<Ea!I<; `$c;I8K?<IPFLJG<CC<;@KFLKa(< `!@E< *9A<:K@JQ8N8PFB@<8E;EFKO@GGFa!I<; `D8Q@E> 4FL:8EK<CC8CCK?8KALJK9PCFFB@E>a

http://lesswrong.com/lw/ns/empty_labels/  +8><b/?<I>LD<EK=IFDFDDFE0J8><c

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 243/309

$DGI<JJ@M< @JEcK @K '<KcJ @EM<EK <M<E DFI< E<N NFI;J `FCFa@J 8E; PFB@< `DLEa @J 8E; O@GGF 8E; `D<IC8:;FE@8Ea @J9FCF8E;DLE

+F@EKC<JJCP :FE=LJ@E> $ K?@EB JF KFF '<KcJ I<GC8:< K?< C89<CJ N@K?K?<;<=@E@K@FEJ

`58N88E;a9<:FD<Jjk`FCF8E;a9<:FD<Jjjkk`(<IC8:;FE@8Ea9<:FD<JjjkkjjSkk

 E; K?< K?@E> KF I<D<D9<I 89FLK K?< I@JKFK<C@8E @;<8 F= :8K<e >FI@<J @J K?8K j k @J K?< entire @E=FID8K@FE F= `Q8N8a $KcJ EFK

ALJK K?8K $ :8E M8IP K?< C89<C 9LK K?8K $ :8E ><K 8CFE> ALJK =@E< N@K?eFLK 8EP C89<C 8K 8CCgK?< ILC<J =FI I@JKFK<C@8E :C8JJ<J NFIB GLI<CP FE JKIL:KLI<J C@B< j k /F :8CC FE< F= K?<J< JKIL:KLI<J `Q8N8aFI 8KK8:? 8EP FK?<I C89<C KF @K @J 8 ?LD8E :FEM<E@<E:< hFI @E:FEM<eE@<E:<i N?@:? D8B<J EFK K?< JC@>?K<JK ;@==<I<E:< KF K?< I@JKFK<C@8EILC<J

'<KcJ J8P K?8K `?LD8Ea @J KF 9< ;<=@E<; 8J 8 DFIK8C =<8K?<IC<JJ9@G<; /?<EK?< :C8JJ@:JPCCF>@JD NFLC;?8M<K?<=FID

 CCjDFIK8CS=<8K?<IJ9@G<;8Ck8I<DFIK8C.F:I8K<J@J8jDFIK8CS=<8K?<IJ9@G<;8Ck/?<I<=FI<.F:I8K<J@JDFIK8C

/?< =<8K F= I<8JFE@E> CFFBJ 8 CFK C<JJ @DGI<JJ@M< EFN ;F<JEcK @K

#<I< K?< illusion of inference :FD<J =IFD K?< C89<CJ N?@:? :FEe:<8C K?< GI<D@J<J 8E; GI<K<E; KF EFM<CKP @E K?< :FE:CLJ@FE -<eGC8:@E> C89<CJ N@K? ;<=@E@K@FEJ I<M<8CJ K?< @CCLJ@FE D8B@E> M@J@9C<

K?< K8LKFCF>PcJ <DG@I@:8C LE?<CG=LCE<JJ 4FL :8E E<M<I J8P K?8K.F:I8K<J @J 8 jDFIK8C S=<8K?<IJ 9@G<;k LEK@C PFL ?8M< F9J<IM<; ?@DKF9<DFIK8C

/?<I<cJ 8E @;<8 N?@:? PFL D8P ?8M< EFK@:<; $ ?8K< K?8K `PFL:8E ;<=@E< 8 NFI; 8EP N8P PFL C@B<a /?@J @;<8 :8D< =IFD K?< I@JeKFK<C@8E EFK@FE F= :8K<>FI@<J J@E:< @= PFL =FCCFN K?< I@JKFK<C@8EILC<J exactly 8E; without flaw gN?@:? ?LD8EJ E<M<I ;F I@JKFe

+8><b/?<+8I89C<F=#<DCF:Bc

+8><b/?<+8I89C<F=#<DCF:Bc +8><b2FI;J8J#@;;<E$E=<I<E:<Jc

%(,1$$+

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 244/309

KC< BE<N G<I=<:KCP N<CC K?8K .F:I8K<J N8J ?LD8E <M<E K?FL>? K?8K N8JEcK ALJK@=@<; LE;<I ?@J ILC<Jg9LK if JFD< @D8>@E8IP EFE?LD8E<EK@KP N<I< KF =FCCFN K?< ILC<J <O8:KCP K?<P NFLC; E<M<I 8II@M< 8K

8 :FEKI8;@:K@FE /?<P NFLC;EcK 8II@M< 8K DL:? F= 8EPK?@E> K?<P :FLC;EcK J8P K?8K .F:I8K<J @J 8 jDFIK8C S=<8K?<IJ 9@G<;k LEK@C K?<P F9J<IM<;?@DKF9<DFIK8C

LK @KcJ EFK JF DL:? K?8K C89<CJ 8I< arbitrary @E K?< I@JKFK<C@8EJPJK<D 8J K?8K K?< I@JKFK<C@8E JPJK<D NFIBJ =@E< N@K?FLK any labels at all  g@K :I8EBJ FLK <O8:KCP K?< J8D< JKI<8D F= K8LKFCF>@<J K?<P ALJKCFFB 8 CFK C<JJ @DGI<JJ@M< /?< C89<CJ 8I< FECP K?<I< KF :I<8K< K?< il- lusion F=@E=<I<E:<

.F @= PFLcI< >F@E> KF ?8M< 8E I@JKFK<C@8E GIFM<I9 8K 8CC K?<

GIFM<I9 J?FLC; 9< EFK `$ :8E ;<=@E< 8 NFI; 8EP N8P $ C@B<a EFI<M<E `<=@E@E> 8 NFI; E<M<I ?8J 8EP :FEJ<HL<E:<Ja 9LK I8K?<I`<=@E@K@FEJ;FEcKE<<;NFI;Ja

-%&6+-!,'/'*+

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 245/309

15. Taboo Your Words

Followup to DGKP'89<CJ

$E K?< >8D< /89FF h9P #8J9IFi K?< F9A<:K@M< @J =FI 8 GC8P<I KF?8M< K?<@I G8IKE<I >L<JJ 8 NFI; NI@KK<E FE 8 :8I; N@K?FLK LJ@E> K?8K NFI; FI =@M< 8;;@K@FE8C NFI;J C@JK<; FE K?< :8I; !FI <O8DGC<

 PFL D@>?K ?8M< KF ><K PFLI G8IKE<I KF J8P `98J<98CCa N@K?FLK LJ@E> K?< NFI;J `JGFIKa `98Ka `?@Ka `G@K:?a `98J<a FI F= :FLIJ< `98J<e98CCa

/?< <O@JK<E:< F= K?@J >8D< JLIGI@J<; D< N?<E $ ;@J:FM<I<; @K2?P NFLC;EcK PFL ALJK J8P `E 8IK@=@:@8C >IFLG :FE=C@:K @E N?@:? PFL

LJ< 8 CFE> NFF;<E :PC@E;<I KF N?8:B 8 K?IFNE JG?<IF@; 8E; K?<EILE9<KN<<E=FLIJ8=<GFJ@K@FEJa

LK K?<E 9P K?< K@D< $ ;@J:FM<I<; K?< >8D< $c; 8CI<8;P 9<<EGI8:K@:@E>@K=FIP<8IJg8C9<@KN@K?8;@==<I<EKGLIGFJ<

4<JK<I;8P N< J8N ?FN I<GC8:@E> K<IDJ N@K? ;<=@E@K@FEJ :FLC;I<M<8C K?< <DG@I@:8C LEGIF;L:K@M@KP  F= K?< :C8JJ@:8C I@JKFK<C@8E JPCeCF>@JD

 CCjDFIK8CS=<8K?<IJ9@G<;k8I<DFIK8C

.F:I8K<J@J8jDFIK8CS=<8K?<IJ9@G<;k/?<I<=FI<.F:I8K<J@JDFIK8C

LKK?<GI@E:@GC<8GGC@<JDL:?DFI<9IF8;CP

 C9<IK `KI<<=8CC@E>@E8;<J<IK<;=FI<JKD8B<J8 JFLE;a8IIP `KI<<=8CC@E>@E8;<J<IK<;=FI<JK;F<JEFKD8B<8JFLE;a

C<8ICP J@E:< FE< J8PJ `JFLE;a 8E; FE< J8PJ `SJFLE;a N< DLJK?8M< 8 :FEKI8;@:K@FE I@>?K LK JLGGFJ< K?8K K?<P 9FK? ;<I<=<Ie<E:<K?<@IGF@EK<IJ9<=FI<JG<8B@E>

 C9<IK `KI<<=8CC@E>@E8;<J<IK<;=FI<JKD8K:?<JjD<D9<IJ?@GK<JKK?@J<M<EK><E<I8K<J8:FLJK@:

http://lesswrong.com/lw/nu/taboo_your_words/ 

+8><bDGKP'89<CJc +8><b/?<+8I89C<F=#<DCF:Bc

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 246/309

 M@9I8K@FEJka8IIP `KI<<=8CC@E>@E8;<J<IK<;=FI<JK;F<JEFKD8K:?jD<D9<IJ?@GK<JKK?@J<M<EK><E<I8K<J8L;@KFIP 

<OG<I@<E:<Jka

)FN K?<I< @J EF CFE><I 8E 8GG8I<EK :FCC@J@FEg8CC K?<P ?8; KF;F N8J GIF?@9@K K?<DJ<CM<J =IFD LJ@E> K?< NFI; sound  $= `8:FLJK@:

 M@9I8K@FEJa :8D< @EKF ;@JGLK< N< NFLC; ALJK GC8P /89FF 8>8@E 8E;J8P `GI<JJLI< N8M<J @E 8 D8K<I@8C D<;@LDa @= E<:<JJ8IP N< NFLC;GC8P /89FF 8>8@E FE K?< NFI; `N8M<a 8E; I<GC8:< @K N@K? K?< N8M<<HL8K@FE h+C8P /89FF FE `8L;@KFIP <OG<I@<E:<a 8E; PFL ><K `/?8K=FID F= J<EJFIP GIF:<JJ@E> N@K?@E K?< ?LD8E 9I8@E N?@:? K8B<J 8J

@EGLK8C@E<8IK@D<J<I@<JF==I<HL<E:PD@O<JaiLK JLGGFJ< FE K?< FK?<I ?8E; K?8K C9<IK 8E; 8IIP N<I< KF

?8M<K?<8I>LD<EK

 C9<IK `.F:I8K<JD8K:?<JK?<:FE:<GKjD<D9<IJ?@GK<JKK?@JG<IJFEN@CC;@<8=K<I;I@EB@E>?<DCF:Bka8IIP `.F:I8K<JD8K:?<JK?<:FE:<GKjD<D9<IJ?@GK<JKK?@JG<IJFEN@CCEFK;@<8=K<I;I@EB@E>?<DCF:Bka

)FN C9<IK 8E; 8IIP ?8M< 8 JL9JK8EK@M< :C8J? F= <OG<:K8K@FEJ8 ;@==<I<E:< @E N?8K K?<P 8EK@:@G8K< J<<@E> 8=K<I .F:I8K<J ;I@EBJ?<DCF:B LK K?<P D@>?K EFK EFK@:< K?@J @= K?<P ?8GG<E<; KF LJ<K?<J8D<NFI;`?LD8Ea=FIK?<@I;@==<I<EK:FE:<GKJ

4FL ><K 8 M<IP ;@==<I<EK G@:KLI< F= N?8K G<FGC< 8>I<< FI ;@J8>I<<89FLK ;<G<E;@E> FE N?<K?<I PFL K8B< 8 C89<CcJe<P<eM@<N hC9<IKJ8PJ `JFLE;a 8E; 8IIP J8PJ `EFK JFLE;a JF K?<P DLJK ;@J8>I<<i FIK8B@E> K?< K<JKcJe<P<eM@<N hC9<IKcJ D<D9<IJ?@G K<JK @J 8:FLJK@: M@e

9I8K@FEJ8IIPcJ@J8L;@KFIP<OG<I@<E:<i"<K KF><K?<I 8 G8:B F=  soi-disant =LKLI@JKJ 8E; 8JB K?<D @= K?<P 9<C@<M< N<cCC ?8M< IK@=@:@8C $EK<CC@><E:< @E K?@IKP P<8IJ 8E; $ NFLC;

 >L<JJ K?8K 8K C<8JK ?8C= F= K?<D N@CC J8P P<J $= PFL C<8M< @K 8K K?8KK?<PcCC J?8B< ?8E;J 8E; :FE>I8KLC8K< K?<DJ<CM<J FE K?<@I :FEJ<EeJLJ LK D8B< K?< K<ID `IK@=@:@8C $EK<CC@><E:<a K89FF 8E; 8JBK?<D KF ;<J:I@9< what K?<P <OG<:K KF J<< N@K?FLK <M<I LJ@E> NFI;JC@B< `:FDGLK<IJa FI `K?@EBa 8E; PFL D@>?K =@E; HL@K< 8 :FE=C@:K F= <OG<:K8K@FEJ ?@;@E> LE;<I K?8K =<8KLI<C<JJ JK8E;8I; NFI; '@B<N@J<

+8><b"L<JJ@E>K?</<8:?<IcJ+8JJNFI;c

-%&6+-!,'/'*+

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 247/309

K?8K FK?<I K<ID E; J<< 8CJF .?8E< '<>>cJ :FDG@C8K@FE F= ;<=@eE@K@FEJF=`@EK<CC@><E:<a

/?< @CCLJ@FE F= LE@KP 8:IFJJ I<C@>@FEJ :8E 9< ;@JG<CC<; 9P D8B@E> 

K?< K<ID `"F;a K89FF 8E; 8JB@E> K?<D KF J8P N?8K @K @J K?<P 9<C@<M<@E FI D8B@E> K?< NFI; `=8@K?a K89FF 8E; 8JB@E> K?<D N?P K?<P 9<eC@<M< @K /?FL>? DFJKCP K?<P NFEcK 9< 89C< KF 8EJN<I 8K 8CC 9<:8LJ<@K @J DFJKCP GIF=<JJ@FE @E K?< =@IJK GC8:< 8E; PFL :8EEFK :F>E@K@M<CP QFFD@EFE8E8L;@FI<:FI;@E>

2?<E PFL =@E; PFLIJ<C= @E G?@CFJFG?@:8C ;@==@:LCK@<J K?< =@IJKC@E< F= ;<=<EJ< @J EFK KF ;<=@E< PFLI GIF9C<D8K@: K<IDJ 9LK KF J<<

 N?<K?<I PFL :8E K?@EB N@K?FLK LJ@E> K?FJ< K<IDJ 8K 8CC *I 8EP F= 

K?<@I J?FIK JPEFEPDJ E; 9< :8I<=LC EFK KF C<K PFLIJ<C= @EM<EK 8 E<N NFI; KF LJ< @EJK<8; <J:I@9< FLKN8I; F9J<IM89C<J 8E; @EK<I@eFI D<:?8E@JDJ ;FEcK LJ< 8 J@E>C< ?8E;C< N?8K<M<I K?8K ?8E;C< D8P 9<

 C9<IK J8PJ K?8K G<FGC< ?8M< `=I<< N@CCa 8IIP J8PJ K?8K G<FGC<;FEcK ?8M< `=I<< N@CCa 2<CC K?8K N@CC :<IK8@ECP ><E<I8K< 8E 8GG8I<EK:FE=C@:K (FJK G?@CFJFG?<IJ NFLC; 8;M@J< C9<IK 8E; 8IIP KF KIP KF ;<=@E< <O8:KCP N?8K K?<P D<8E 9P `=I<< N@CCa FE N?@:? KFG@: K?<P 

 N@CC :<IK8@ECP 9< 89C< KF ;@J:FLIJ< 8K >I<8K C<E>K? $ NFLC; 8;M@J<

 C9<IK 8E; 8IIP KF ;<J:I@9< N?8K @K @J K?8K K?<P K?@EB G<FGC< ;F FI;F EFK ?8M< N@K?FLK LJ@E> K?< G?I8J< `=I<< N@CCa 8K 8CC h$= PFL N8EKKF KIP K?@J 8K ?FD< PFL J?FLC; 8CJF 8MF@; K?< NFI;J `:?FFJ<a `8:Ka`;<:@;<a`;<K<ID@E<;a`I<JGFEJ@9C<aFI8EPF=K?<@IJPEFEPDJi

/?@J @J FE< F= K?< EFEJK8E;8I; KFFCJ @E DP KFFC9FO 8E; @E DP ?LD9C< FG@E@FE @K NFIBJ way way 9<KK<I K?8E K?< JK8E;8I; FE< $K8CJFI<HL@I<JDFI<<==FIKKFLJ<PFL><KN?8KPFLG8P=FI

http://www.singinst.org/blog/2007/09/30/three-major-singularity-schools/ 

http://arxiv.org/abs/0706.3639 +8><b<C@<=@E<C@<=c

,''1'-* /'*+

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 248/309

16. Replace the Symbol with the Substance

Continuation of  /89FF4FLI2FI;J

Followup to *I@>@E8C.<<@E>  'FJK+LIGFJ<J

2?8K ;F<J @K K8B< KFg8J @E P<JK<I;8PcJ <O8DGC<gJ<< 8 `98J<98CC >8D<a 8J `E 8IK@=@:@8C >IFLG :FE=C@:K @E N?@:? PFL LJ< 8 CFE> NFF;e<E :PC@E;<I KF N?8:B 8 K?IFNE JG?<IF@; 8E; K?<E ILE 9<KN<<E =FLIJ8=< GFJ@K@FEJa 2?8K ;F<J @K K8B< KF GC8P K?< I8K@FE8C@JK M<IJ@FE F= /89FF @E N?@:? K?< >F8C @J EFK KF =@E; 8 JPEFEPD K?8K @JEcK FE K?<:8I; 9LK KF =@E; 8 N8P F= ;<J:I@9@E> N@K?FLK K?< JK8E;8I; :FE:<GKe

?8E;C<4FL ?8M< KF M@JL8C@Q< 4FL ?8M< KF D8B< PFLI D@E;cJ <P< J<<

K?< ;<K8@CJ 8J K?FL>? CFFB@E> =FI K?< =@IJK K@D< 4FL ?8M< KF G<Ie=FID8E *I@>@E8C.<<@E> 

$J K?8K 8 `98Ka )F @KcJ 8 CFE> IFLE; K8G<I@E> NFF;<E IF;E8IIFN@E>8KFE<<E;JFK?8K8?LD8E:8E>I8JG8E;JN@E>@K

$J K?8K 8 `98CCa )F @KcJ 8 C<8K?<Ie:FM<I<; JG?<IF@; N@K? 8 JPDeD<KI@:8C JK@K:?@E> G8KK<IE ?8I; 9LK EFK D<K8Ce?8I; N?@:? JFD<FE<:8E>I8JG8E;K?IFNFIJKI@B<N@K?K?<NFF;<EIF;FI:8K:?

 I< K?FJ< `98J<Ja )F K?<PcI< =@O<; GFJ@K@FEJ FE 8 >8D< =@<C;K?8K GC8P<IJ KIP KF ILE KF 8J HL@:BCP 8J GFJJ@9C< 9<:8LJ< F= K?<@I J8=<eKPN@K?@EK?<>8D<cJ8IK@=@:@8CILC<J

/?< :?@<= F9JK8:C< KF G<I=FID@E> 8E FI@>@E8C J<<@E> @J K?8K PFLID@E; 8CI<8;P ?8J 8 E@:< E<8K JLDD8IP 8 E@:< C@KKC< <8JPeKFeLJ< :FEe:<GK ?8E;C< '@B< K?< NFI; `98J<98CCa FI `98Ka FI `98J<a $K K8B<J8E <==FIK KF JKFG PFLI D@E; =IFD JC@;@E> ;FNE K?< =8D@C@8I G8K? K?<<8JP G8K? K?< G8K? F= C<8JK I<J@JK8E:< N?<I< K?< JD8CC =<8KLI<C<JJ

 NFI; ILJ?<J @E 8E; F9C@K<I8K<J K?< ;<K8@CJ PFLcI< KIP@E> KF J<<  NFI; @KJ<C= :8E ?8M< K?< ;<JKIL:K@M< =FI:< F= :C@:?< 8 NFI; @KJ<C= :8E:8IIPK?<GF@JFEF=8 :8:?<;K?FL>?K

http://lesswrong.com/lw/nv/replace_the_symbol_with_the_substance/  +8><b/89FF4FLI2FI;Jc +8><b*I@>@E8C.<<@E>c http://lesswrong.com/lw/le/lost_purposes/  +8><b*I@>@E8C.<<@E>c

+8><b-8K@FE8C@KP8E;K?<E>C@J?'8E>L8><c +8><b8:?<;/?FL>?KJc

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 249/309

+C8P@E> K?< >8D< F= /89FF g9<@E> 89C< KF ;<J:I@9< N@K?FLKLJ@E> K?< JK8E;8I; GF@EK<IC89<C?8E;C<g@J FE< F= K?< fundamental I8K@FE8C@JK :8G8:@K@<J $K F::LG@<J K?< J8D< GI@DFI;@8C C<M<C 8J K?<

?89@K F= :FEJK8EKCP 8JB@E> `2?Pa FI `2?8K ;F<J K?@J 9<C@<= D8B<D<8EK@:@G8K<a

/?<8IK@J:CFJ<CPI<C8K<;KF

[ +I8>D8K@JD9<:8LJ<J<<@E>@EK?@JN8PF=K<E>@M<JPFL8 DL:?:CFJ<I:FEE<:K@FEKF 8EK@:@G8K<;<OG<I@<E:<I8K?<IK?8E GIFGFJ@K@FE8C9<C@<= 

[ -<;L:K@FE@JD9<:8LJ<J<<@E>@EK?@JN8PF=K<E=FI:<JPFLKF;IFG;FNEKF8CFN<IC<M<CF=FI>8E@Q8K@FECFFB8KK?<

G8IKJ@EJK<8;F=PFLI<P<JB@GG@E>FM<IK?<N?FC<[ #L>>@E>K?<HL<IP 9<:8LJ<NFI;JF=K<E;@JKI8:KPFL=IFDK?<HL<JK@FEPFLI<8CCPN8EKKF8JB

[ MF@;@E> :8:?<;K?FL>?KJN?@:?N@CCILJ?@ELJ@E> JK8E;8I;NFI;JJFPFL:8E9CF:BK?<D9PK89FF@E> JK8E;8I;NFI;J

[ /?<NI@K<IcJILC<F=`.?FN;FEcKK<CCaN?@:??8JGFN<I8DFE>I8K@FE8C@JKJ

[ E; EFKCFJ@E>J@>?KF=PFLIFI@>@E8CGLIGFJ<

#FN:FLC;K89FF@E>8NFI;?<CGPFLB<<GPFLIGLIGFJ<!IFD 'FJK+LIGFJ<J

 JPFLI<8;K?@JJFD<PFLE>D8EFINFD8E@JJ@KK@E>8K8;<JB@E8LE@M<IJ@KP<8IE<JKCPJKL;P@E>D8K<I@8CK?<P ?8M<EF@EK<EK@FEF=<M<ILJ@E>8E;EF@EK<I<JK@EBEFN@E>=FI@KJFNEJ8B< /?<PN8EK8?@>?eG8P@E>AF98E;K?<?@>?eG8P@E>AF9I<HL@I<J8G@<:<F=G8G<I8E;K?<G@<:<F=G8G<II<HL@I<J8GI<M@FLJD8JK<IcJ;<>I<<8E;K?<D8JK<IcJ;<>I<<I<HL@I<J898:?<CFIcJ;<>I<<8E;K?<LE@M<IJ@KPK?8K>I8EKJK?<98:?<CFIcJ;<>I<<I<HL@I<JPFLKFK8B<8:C8JJ@EK?e:<EKLIPBE@KK@E>G8KK<IEJKF

 >I8;L8K< .FK?<P;@C@><EKCPJKL;P@EK<E;@E>KF=FI><K@K

+8><b/89FF4FLI2FI;Jc +8><b(8B@E><C@<=J+8P-<EKh@EEK@:@G8K<;OG<I@<E:<Jic

+8><b<C@<=@E<C@<=c +8><b#L>K?<,L<IPc +8><b8:?<;/?FL>?KJc

http://lesswrong.com/lw/le/lost_purposes/  http://lesswrong.com/lw/le/lost_purposes/ 

* ($,+1%'$/!,,+-+,&

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 250/309

8CCK?<DFD<EKK?<=@E8C<O8D@J8;D@E@JK<I<;9LKJK@CCJ<I@FLJCPNFIB@E>8N8P9<:8LJ<K?<P want K?8KG@<:<F= G8G<I

2?P 8I< PFL >F@E> KF `J:?FFCa /F ><K 8E `<;L:8K@FEa <E;@E> @E 8 `;<>I<<a C8EB FLK K?< =FI9@;;<E NFI;J 8E; 8CC K?<@I F9M@FLJJPEFEPDJ M@JL8C@Q< K?< 8:KL8C ;<K8@CJ 8E; PFLcI< DL:? DFI< C@B<CP KF EFK@:< K?8K `J:?FFCa :LII<EKCP J<<DJ KF :FEJ@JK F= J@KK@E> E<OKKF 9FI<; K<<E8><IJ C@JK<E@E> KF D8K<I@8C PFL 8CI<8;P BEFN K?8K 8 `;<>I<<a @J 8 G@<:< F= G8G<I N@K? JFD< NI@K@E> FE @K 8E; K?8K `<;Le:8K@FEa@J=FI><KK@E>K?<D8K<I@8C8JJFFE8JPFLcI<K<JK<;FE@K

'<8BP ><E<I8C@Q8K@FEJ F=K<E D8E@=<JK K?IFL>? :8K<>FI@Q8K@FEJ

+<FGC< N?F 8:KL8CCP C<8IE @E :C8JJIFFDJ 8I< :8K<>FI@Q<; 8J `><KK@E> 8E <;L:8K@FEa JF `><KK@E> 8E <;L:8K@FEa DLJK 9< >FF; 9LK K?<E8EPFE< N?F 8:KL8CCP J?FNJ LG 8K 8 :FCC<>< N@CC 8CJF D8K:? 8>8@EJKK?<:FE:<GK`><KK@E>8E<;L:8K@FEaN?<K?<IFIEFKK?<PC<8IE

.KL;<EKJ N?F LE;<IJK8E; D8K? N@CC ;F N<CC FE K<JKJ 9LK @=  PFL I<HL@I< J:?FFCJ KF GIF;L:< >FF; K<JK J:FI<J K?<PcCC JG<E; 8CCK?<@I K@D< K<8:?@E> KF K?< K<JK mental category, K?8K @DG<I=<:KCP D8K:?<J PFLI >F8C :8E GIF;L:< K?< J8D< B@E; F= @E:<EK@M< =8@CLI<

internally. 4FL N8EK KF C<8IE JF PFL E<<; 8E `<;L:8K@FEa 8E; K?<E8J CFE> 8J PFLcI< ><KK@E> 8EPK?@E> K?8K D8K:?<J 8>8@EJK K?< :8K<>FIP `<;L:8K@FEa PFL D8P EFK EFK@:< N?<K?<I PFLcI< C<8IE@E> FI EFK *I

 PFLcCC EFK@:< 9LK PFL NFEcK I<8C@Q< PFLcM< CFJK J@>?K F= PFLI FI@>@E8CGLIGFJ< 9<:8LJ< PFLcI< `><KK@E> 8E <;L:8K@FEa 8E; K?8KcJ ?FN PFLD<EK8CCP;<J:I@9<;PFLI>F8C

/F :8K<>FI@Q< @J KF K?IFN 8N8P @E=FID8K@FE $= PFLcI< KFC; K?8K8 =8CC@E> KI<< D8B<J 8 `JFLE;a PFL ;FEcK BEFN N?8K K?< 8:KL8CJFLE; @J PFL ?8M<EcK 8:KL8CCP ?<8I; K?< KI<< =8CC@E> $= 8 :F@E C8E;J

`?<8;Ja PFL ;FEcK BEFN @KJ I8;@8C FI@<EK8K@FE 9CL< <>>eJ?8G<;K?@E> D8P 9< 8 `9C<>>a 9LK N?8K @= K?< <O8:K <>> J?8G< M8I@<J FI K?<<O8:K J?8;< F= 9CL< 4FL N8EK KF LJ< :8K<>FI@<J KF K?IFN 8N8P @IeI<C<M8EK @E=FID8K@FE KF J@=K >FC; =IFD ;LJK 9LK F=K<E K?< JK8E;8I;:8K<>FI@Q8K@FE <E;J LG K?IFN@E> FLK I<C<M8EK @E=FID8K@FE KFF E;

 N?<E PFL <E; LG @E K?8K JFIK F= D<EK8C KIFL9C< K?< =@IJK 8E; DFJKF9M@FLJJFCLK@FE@JKFGC8P/89FF

!FI <O8DGC< `+C8P /89FFa @J @KJ<C= 8 C<8BP ><E<I8C@Q8K@FE #8Je9IFcJ M<IJ@FE @J EFK K?< I8K@FE8C@JK M<IJ@FE K?<P FECP C@JK =@M< 8;e

http://lesswrong.com/lw/lc/leaky_generalizations/ 

-%&6+-!,'/'*+

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 251/309

;@K@FE8C 98EE<; NFI;J FE K?< :8I; 8E; K?8KcJ EFK E<8ICP <EFL>?:FM<I8>< KF <O:CL;< K?@EB@E> @E =8D@C@8I FC; NFI;J 2?8K I8K@FeE8C@JKJ ;F NFLC; :FLEK 8J GC8P@E> /89FFg@K NFLC; D8K:? 8>8@EJK

K?< `GC8P /89FFa :FE:<GKg9LK EFK <M<IPK?@E> K?8K :FLEKJ 8J GC8Pe@E> /89FF NFIBJ KF =FI:< FI@>@E8C J<<@E> $= PFL ALJK K?@EB `GC8P /89FF KF =FI:< FI@>@E8C J<<@E>a PFLcCC JK8IK K?@EB@E> K?8K 8EPK?@E> K?8K:FLEKJ8JGC8P@E>/89FFDLJK:FLEK8JFI@>@E8CJ<<@E>

/?< I8K@FE8C@JK M<IJ@FE @JEcK 8 >8D< N?@:? D<8EJ K?8K PFL :8EcK N@E 9P KIP@E> KF 9< :C<M<I 8E; JKI<K:?@E> K?< ILC<J 4FL ?8M< KF GC8P /89FF N@K? 8 MFCLEK8IP ?8E;@:8G .KFG PFLIJ<C= =IFD LJ@E> JPEeFEPDJ K?8K 8I<EcK FE K?< :8I; 4FL 8CJF ?8M< KF JKFG PFLIJ<C= =IFD@EM<EK@E> 8 E<N J@DGC< NFI; FI G?I8J< K?8K =LE:K@FEJ 8J 8E <HL@Me

8C<EK D<EK8C ?8E;C< KF K?< FC; FE< 4FL 8I< KIP@E> KF QFFD @E FE PFLI D8G EFK I<E8D< K?< :@K@<J ;<I<=<I<E:< K?< GF@EK<I EFK 8CCFe:8K< 8 E<N GF@EK<I J<< K?< <M<EKJ 8J K?<P ?8GG<E EFK I<NI@K< K?<:C@:?<@E8;@==<I<EKNFI;@E>

P M@JL8C@Q@E> K?< GIF9C<D @E DFI< ;<K8@C PFL :8E J<< K?< CFJKGLIGFJ< O8:KCP N?8K ;F PFL ;F N?<E PFL `GC8P /89FFa 2?8KGLIGFJ<;F<J<8:?8E;<M<IPG8IKJ<IM<

$= PFL J<< PFLI 8:K@M@K@<J 8E; J@KL8K@FE FI@>@E8CCP PFL N@CC 9< 89C<

KF FI@>@E8CCP J<< PFLI >F8CJ 8J N<CC $= PFL :8E CFFB N@K? =I<J? <P<J8J K?FL>? =FI K?< =@IJK K@D< PFL N@CC J<< PFLIJ<C= ;F@E> K?@E>J K?8K PFLNFLC;E<M<I;I<8DF=;F@E>@=K?<PN<I<EFK?89@KJ

+LIGFJ< @J CFJK N?<E<M<I K?< JL9JK8E:< hC<8IE@E> BEFNC<;><?<8CK?i @J ;@JGC8:<; 9P K?< JPD9FC h8 ;<>I<< 8 K<JK J:FI< D<;@:8C:8I<i /F ?<8C 8 CFJK GLIGFJ< FI 8 CFJJP :8K<>FI@Q8K@FE PFL DLJK ;FK?<I<M<IJ<

-<GC8:< K?< JPD9FC N@K? K?< JL9JK8E:< I<GC8:< K?< J@>E@=@<I N@K? K?< J@>E@=@<; I<GC8:< K?< GIFG<IKP N@K? K?< D<D9<IJ?@G K<JK

I<GC8:< K?< NFI; N@K? K?< D<8E@E> I<GC8:< K?< C89<C N@K? K?<:FE:<GK I<GC8:< K?< JLDD8IP N@K? K?< ;<K8@CJ I<GC8:< K?< GIFOP HL<JK@FE N@K? K?< I<8C HL<JK@FE ;<I<=<I<E:< K?< GF@EK<I ;IFG @EKF8 CFN<I C<M<C F= FI>8E@Q8K@FE D<EK8CCP J@DLC8K< K?< GIF:<JJ @EJK<8;F=E8D@E>@KQFFD@EFEPFLID8G

`/?< .@DGC< /ILK?a N8J ><E<I8K<; 9P 8E <O<I:@J< F= K?@J ;@Je:@GC@E< KF ;<J:I@9< `KILK?a FE 8 CFN<I C<M<C F= FI>8E@Q8K@FE N@K?FLK@EMFB@E> K<IDJ C@B< `8::LI8K<a `:FII<:Ka `I<GI<J<EKa `I<=C<:Ka `J<e

http://yudkowsky.net/bayes/truth.html 

* ($,+1%'$/!,,+-+,&

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 252/309

D8EK@:a `9<C@<M<a `BEFNC<;><a `D8Ga FI `I<8Ca hE; I<D<D9<IK?8K K?< >F8C @J EFK really KF GC8P /89FFgK?< NFI; `KIL<a 8GG<8IJ@E K?< K<OK 9LK not  KF ;<=@E< KILK? $K NFLC; ><K 8 9LQQ<I @E #8Je

9IFcJ >8D< 9LK N<cI< EFK actually GC8P@E> K?8K >8D< JB PFLIJ<C=  N?<K?<I K?< ;F:LD<EK =LC=@CC<; @KJ GLIGFJ< EFK N?<K?<I @K =FCCFN<;K?<ILC<Ji

8P<JcJ -LC< @KJ<C= ;<J:I@9<J `<M@;<E:<a @E GLI< D8K? N@K?FLKLJ@E> NFI;J C@B< `@DGC@<Ja `D<8EJa `JLGGFIKJa `GIFM<Ja FI `ALJK@e=@<Ja .<K FLK KF define JL:? G?@CFJFG?@:8C K<IDJ 8E; PFLcCC ALJK >F@E:@I:C<J

 E; K?<E K?<I<cJ K?< DFJK @DGFIK8EK NFI; F= 8CC KF /89FF $cM<F=K<E  N8IE<; K?8K PFL J?FLC; 9< :8I<=LC EFK KF FM<ILJ< @K FI<M<E 8MF@; K?< :FE:<GK @E :<IK8@E :8J<J )FN PFL BEFN K?< I<e8C I<8JFE N?P $KcJ EFK 8 98; JL9A<:K KF K?@EB 89FLK LK PFLI KIL<LE;<IJK8E;@E> @J D<8JLI<; 9P PFLI 89@C@KP KF ;<J:I@9< N?8K PFLcI<;F@E>8E;N?P without LJ@E>K?8KNFI;FI8EPF=@KJJPEFEPDJ

http://yudkowsky.net/virtues/ 

+8><b/NFLCK&F8EJc http://lesswrong.com/lw/nc/newcombs_problem_and_regret_of_rationality/ 

-%&6+-!,'/'*+

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 253/309

17. Fallacies of Compression 

Followup to -<GC8:<K?<.PD9FCN@K?K?<.L9JK8E:<

`/?< D8G @J EFK K?< K<II@KFIPa 8J K?< J8P@E> >F<J /?< FECP C@=<eJ@Q< 8KFD@:8CCP ;<K8@C<; n 8::LI8K< D8G F= 8C@=FIE@8 @J8C@=FIE@8 LK 8C@=FIE@8 ?8J @DGFIK8EK I<>LC8I@K@<J JL:? 8J K?<J?8G< F= @KJ ?@>?N8PJ K?8K :8E 9< ;<J:I@9<; LJ@E> M8JKCP C<JJ @E=FIeD8K@FEgEFK KF D<EK@FE M8JKCP C<JJ physical material  gK?8E @K NFLC;K8B< KF ;<J:I@9< <M<IP 8KFD N@K?@E K?< JK8K< 9FI;<IJ #<E:< K?<other J8P@E> `/?< D8G @J EFK K?< K<II@KFIP 9LK PFL :8EcK =FC; LG K?<K<II@KFIP8E;GLK@K@EPFLI>CFM<:FDG8IKD<EKa

  G8G<I D8G F= 8C@=FIE@8 8K 8 J:8C< F= B@CFD<K<IJ KF :<EeK@D<K<I h8 D@CC@FE KF FE<i ;F<JEcK ?8M< IFFD KF J?FN K?< ;@JK@E:KGFJ@K@FE F= KNF =8CC<E C<8M<J CP@E> 8 :<EK@D<K<I 8G8IK FE K?< J@;<e

 N8CB M<E @= K?< D8G KI@<; KF J?FN K?< C<8M<J K?< C<8M<J NFLC;8GG<8I 8J K?< J8D< GF@EK FE K?< D8G FI I8K?<I K?< D8G NFLC; E<<;8 =<8KLI< J@Q< F= E8EFD<K<IJ N?@:? @J 8 =@E<I I<JFCLK@FE K?8EDFJK9FFBGI@EK<IJ?8E;C<EFKKFD<EK@FE?LD8E<P<J

-<8C@KP @J M<IP C8I><gALJK K?< G8IK N< :8E J<< @J 9@CC@FEJ F= C@>?KP<8IJ 8:IFJJ LK PFLI D8G F= I<8C@KP @J NI@KK<E FE 8 =<N GFLE;J

F= E<LIFEJ =FC;<; LG KF =@K @EJ@;< PFLI JBLCC $ ;FEcK D<8E KF 9< @EeJLCK@E>9LKPFLIJBLCC@JK@EP:FDG8I8K@M<CPJG<8B@E>

$E<M@K89CP K?<E :<IK8@E K?@E>J K?8K 8I< ;@JK@E:K @E I<8C@KP N@CC9<:FDGI<JJ<;@EKFK?<J8D<GF@EKFEPFLID8G

LK N?8K K?@J =<<CJ C@B< =IFD @EJ@;< @J EFK K?8K PFL J8P `*?CFFB $cD :FDGI<JJ@E> KNF K?@E>J @EKF FE< GF@EK FE DP D8Ga2?8K @K feels C@B< =IFD @EJ@;< @J K?8K K?<I< @J ALJK one K?@E> 8E; PFL8I<J<<@E>@K

  JL==@:@<EKCP PFLE> :?@C; FI 8 JL==@:@<EKCP 8E:@<EK "I<<BG?@CFJFG?<I NFLC; EFK BEFN K?8K K?<I< N<I< JL:? K?@E>J 8J `8:FLJeK@: M@9I8K@FEJa FI `8L;@KFIP <OG<I@<E:<Ja /?<I< NFLC; ALJK 9< 8 J@E>C< K?@E> K?8K ?8GG<E<; N?<E 8 KI<< =<CC 8 J@E>C< <M<EK :8CC<;`JFLE;a

http://lesswrong.com/lw/nw/fallacies_of_compression/ 

+8><b-<GC8:<K?<.PD9FCN@K?K?<.L9JK8E:<c +8><b#FNEC>FI@K?D!<<CJ!IFD$EJ@;<c

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 254/309

/F I<8C@Q< K?8K K?<I< 8I< two ;@JK@E:K <M<EKJ LE;<ICP@E>  oneGF@EK FE PFLI D8G @J 8E <JJ<EK@8CCP scientific :?8CC<E><g8 9@> ;@==@e:LCKJ:@<EK@=@::?8CC<E><

.FD<K@D<J =8CC8:@<J F= :FDGI<JJ@FE I<JLCK =IFD :FE=LJ@E> KNFBEFNE K?@E>J LE;<I K?< J8D< C89<CgPFL BEFN 89FLK 8:FLJK@: M@e9I8K@FEJ 8E; PFL BEFN 89FLK 8L;@KFIP GIF:<JJ@E> @E 9I8@EJ 9LK PFL:8CC K?<D 9FK? `JFLE;a 8E; JF :FE=LJ< PFLIJ<C= LK K?< DFI< ;8Ee

 ><IFLJ =8CC8:P F= :FDGI<JJ@FE 8I@J<J =IFD ?8M@E> no idea whatsoever K?8K KNF ;@JK@E:K <EK@K@<J <M<E exist  /?<I< @J ALJK FE< D<EK8C =FC;<I@E K?< =@C@E> JPJK<D C89<C<; `JFLE;a 8E; <M<IPK?@E> K?FL>?K 89FLK`JFLE;a ;IFGJ @EKF K?8K FE< =FC;<I $KcJ EFK K?8K K?<I< 8I< KNF =FC;e<IJ N@K? K?< J8D< C89<C K?<I<cJ ALJK 8 J@E>C< =FC;<I P ;<=8LCK K?<

D8G @J :FDGI<JJ<; N?P NFLC; K?< 9I8@E :I<8K< KNF D<EK8C 9L:B<KJ N?<I<FE<NFLC;J<IM<

*I K?@EB F= 8 DPJK<IP EFM<C @E N?@:? K?< ;<K<:K@M<cJ :I@K@:8C @EeJ@>?K @J K?8K FE< F= K?< JLJG<:KJ ?8J 8E @;<EK@:8C KN@E $E K?< :FLIJ<F= K?< ;<K<:K@M<cJ FI;@E8IP NFIB ?@J AF9 @J ALJK KF F9J<IM< K?8K8IFC @J N<8I@E> I<; K?8K J?< ?8J 9C8:B ?8@I K?8K ?<I J8E;8CJ 8I<C<8K?<Ig9LK 8CC K?<J< 8I< facts about 8IFC $KcJ <8JP <EFL>? KF HL<JeK@FE 8E @E;@M@;L8C =8:K C@B< 2<8IJ-<;h8IFCi FI C8:B#8@Ih8IFCi(8P9< C8:B#8@Ih8IFCi @J =8CJ< (8P9< 8IFC ;P<J ?<I ?8@I(8P9< IFNE#8@Ih8IFCi LK @K K8B<J 8 JL9KC<I ;<K<:K@M< KF NFEe;<I @= K?< 8IFC @E 2<8IJ-<;h8IFCi 8E; C8:B#8@Ih8IFCigK?<8IFC =@C< @EKF N?@:? ?@J F9J<IM8K@FEJ ;IFGgJ?FLC; 9< JGC@K @EKFtwo =@C<J (8P9< K?<I< 8I< KNF 8IFCJ JF K?8K K?< 8IFC N?F NFI<I<;@JEFKK?<J8D<NFD8E8JK?<8IFCN?F?8;9C8:B?8@I

#<I< @K @J K?< M<IP 8:K F=  creating KNF ;@==<I<EK 9L:B<KJ K?8K @JK?< JKIFB< F= ><E@LJ @EJ@>?K b/@J <8J@<I KF HL<JK@FE FE<cJ =8:KJ K?8EFE<cJFEKFCF>P

/?< D8G F= I<8C@KP :FEK8@E<; @E 8 ?LD8E 9I8@E LEC@B< 8 G8G<ID8G F= 8C@=FIE@8 :8E <OG8E; ;PE8D@:8CCP N?<E N< NI@K< ;FNEDFI< ;<K8@C<; ;<J:I@GK@FEJ LK N?8K K?@J =<<CJ C@B< =IFD @EJ@;<@J EFK JF DL:? QFFD@E> @E FE 8 D8G 8J =@JJ@FE@E> 8E @E;@M@J@9C<8KFDgK8B@E> one thing  h@K =<CK C@B< FE< K?@E>i 8E; JGC@KK@E> @K @EKFKNFFIDFI<K?@E>J

*=K<E K?@J D8E@=<JKJ @E K?< :I<8K@FE F= E<N NFI;J C@B< `8:FLJK@: M@9I8K@FEJa 8E; `8L;@KFIP <OG<I@<E:<Ja @EJK<8; F= ALJK `JFLE;a.FD<K?@E> 89FLK :I<8K@E> K?< E<N E8D< J<<DJ KF 8CCF:8K< K?< E<N 

-%&6+-!,'/'*+

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 255/309

9L:B<K /?< ;<K<:K@M< @J C@89C< KF JK8IK :8CC@E> FE< F= ?@J JLJG<:KJ`8IFCea FI `K?< *K?<I 8IFCa 8CDFJK 8J JFFE 8J ?< I<8C@Q<J K?8KK?<I<8I<KNFF=K?<D

LK <OG8E;@E> K?< D8G @JEcK 8CN8PJ 8J J@DGC< 8J ><E<I8K@E> E<N :@KP E8D<J $K @J 8 JKIFB< F= J:@<EK@=@: @EJ@>?K KF I<8C@Q< K?8K JL:?K?@E>J8J8:FLJK@:M@9I8K@FEJFI8L;@KFIP<OG<I@<E:<J<M<E exist.

/?< F9M@FLJ DF;<IEe;8P @CCLJKI8K@FE NFLC; 9< NFI;J C@B< `@EK<CeC@><E:<a FI `:FEJ:@FLJE<JJa M<IP EFN 8E; K?<E FE< J<<J 8 GI<JJI<C<8J< :C8@D@E> K?8K 8 I<J<8I:? ?8J `<OGC8@E<; :FEJ:@FLJE<JJa 9<e:8LJ< 8 K<8D F= E<LIFCF>@JKJ @EM<JK@>8K<; 8 #Q <C<:KI@:8C I?PK?DK?8K D@>?K ?8M< JFD<K?@E> KF ;F N@K? :IFJJeDF;8C@KP 9@E;@E> F= J<EJFIP @E=FID8K@FE FI 9<:8LJ< K?<P @EM<JK@>8K<; K?< I<K@:LC8I 8:K@e

 M8K@E> JPJK<D K?8K B<<GJ ?LD8EJ 8N8B< /?8KcJ 8E <OKI<D< <O8DeGC< 8E; K?< LJL8C =8@CLI<J 8I< DFI< JL9KC< 9LK K?<P 8I< F= K?< J8D<B@E; /?< G8IK F= `:FEJ:@FLJE<JJa K?8K G<FGC< =@E; DFJK @EK<I<JK@E> @J I<=C<:K@M@KP J<C=e8N8I<E<JJ I<8C@Q@E> K?8K K?< G<IJFE $ J<< @E K?<D@IIFI @J `D<a K?8K 8E; K?< ?8I; GIF9C<D F= JL9A<:K@M< <OG<I@<E:<8J ;@JK@E>L@J?<; 9P ?8CD<IJ 2< 8CJF C89<C `:FEJ:@FLJa K?< JK8K<F= 9<@E> 8N8B< I8K?<I K?8E 8JC<<G @E FLI ;8@CP :P:C< LK K?<P 8I<8CC ;@==<I<EK :FE:<GKJ >F@E> LE;<I K?< J8D< E8D< 8E; K?< LE;<IeCP@E> G?<EFD<E8 8I< ;@==<I<EK J:@<EK@=@: GLQQC<J 4FL :8E <OGC8@E9<@E>8N8B<N@K?FLK<OGC8@E@E>I<=C<:K@M@KPFIJL9A<:K@M@KP

!8CC8:@<J F= :FDGI<JJ@FE 8CJF LE;<IC@< K?< 98@Ke8E;eJN@K:? K<:?eE@HL< @E G?@CFJFG?PgPFL 8I>L< 89FLK `:FEJ:@FLJE<JJa LE;<I FE<;<=@E@K@FE hC@B< K?< 89@C@KP KF K?@EB 89FLK K?@EB@E>i 8E; K?<E 8GGCP K?< :FE:CLJ@FEJ KF `:FEJ:@FLJE<JJa LE;<I 8 ;@==<I<EK ;<=@E@K@FE hC@B<JL9A<:K@M@KPi *= :FLIJ< @K D8P 9< K?8K K?< KNF 8I< K?< J8D< K?@E>9LK @= JF ><EL@E<CP understanding K?@J =8:K NFLC; I<HL@I< first 8 :FEe:<GKL8CJGC@K8E; then 8><E@LJJKIFB<F=I<LE@=@:8K@FE

OG8E;@E> PFLI D8G @J h$ J8P 8>8@Ei 8  scientific :?8CC<E>< G8IKF= K?< 8IK F= J:@<E:< K?< JB@CC F= @EHL@I@E> @EKF K?< NFIC; hE; F= :FLIJ< PFL :8EEFK JFCM< 8 J:@<EK@=@: :?8CC<E>< 9P 8GG<8C@E> KF ;@:K@FeE8I@<J EFI D8JK<I 8 :FDGC<O JB@CC F= @EHL@IP 9P J8P@E> `$ :8E ;<=@E<8 NFI; 8EP N8P $ C@B<ai 2?<I< PFL J<< 8 J@E>C< :FE=LJ@E> K?@E>

 N@K? GIFK<8E 8E; J<C=e:FEKI8;@:KFIP 8KKI@9LK<J @K @J 8 >FF; >L<JJK?8K PFLI D8G @J :I8DD@E> KFF DL:? @EKF FE< GF@EKgPFL E<<; KFGIP @K 8G8IK 8E; 8CCF:8K< JFD< E<N 9L:B<KJ /?@J @J EFK C@B< defining K?< J@E>C< K?@E> PFL J<< 9LK @K does F=K<E =FCCFN =IFD =@>LI@E> FLK

?FNKFK8CB89FLKK?<K?@E>N@K?FLKLJ@E>8J@E>C<D<EK8C?8E;C<

$$!+''%(*++!'&

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 256/309

.F K?< JB@CC F= GIP@E> 8G8IK K?< D8G @J C@EB<; KF K?< I8K@FE8C@JK M<IJ@FE F= /89FF 8E; KF K?< N@J< LJ< F= NFI;J 9<:8LJ< NFI;J F=eK<E I<GI<J<EK K?< GF@EKJ FE FLI D8G K?< C89<CJ LE;<I N?@:? N< =@C<

FLI GIFGFJ@K@FEJ 8E; K?< 9L:B<KJ @EKF N?@:? N< ;IFG FLI @E=FID8eK@FE MF@;@E> 8 J@E>C< NFI; FI 8CCF:8K@E> E<N FE<J @J F=K<E G8IKF=K?<JB@CCF=<OG8E;@E>K?<D8G

+8><b-<GC8:<K?<.PD9FCN@K?K?<.L9JK8E:<c +8><b/89FF4FLI2FI;Jc

-%&6+-!,'/'*+

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 257/309

18. Categorizing Has Consequences

Followup to !8CC8:@<JF=FDGI<JJ@FE

 DFE> K?< D8EP ><E<K@: M8I@8K@FEJ 8E; DLK8K@FEJ PFL :8IIP @E PFLI ><EFD< K?<I< 8I< 8 M<IP =<N 8CC<C<J PFL GIF989CP BEFNg@E:CL;@E> K?FJ< ;<K<ID@E@E> PFLI 9CFF; KPG< K?< GI<J<E:<FI 89J<E:< F= K?< 8E; 8EK@><EJ $= PFL I<:<@M< 8 9CFF;KI8EJ=LJ@FE :FEK8@E@E> 8E 8EK@><E PFL ;FEcK ?8M< @K N@CC KI@>><I 8E8CC<I>@: I<8:K@FE $K N8J &8IC '8E;JK<@E<IcJ ;@J:FM<IP F= K?@J =8:K8E; ?FN KF K<JK =FI :FDG8K@9C< 9CFF; KPG<J K?8K D8;< @K GFJJ@9C<KF KI8EJ=LJ< 9CFF; N@K?FLK B@CC@E> K?< G8K@<EK h )F9<C +I@Q<

@E (<;@:@E<i CJF @= 8 DFK?<I N@K? 9CFF; KPG< h=FI <O8DGC<i9<8IJ 8 :?@C; N@K? 9CFF; KPG< K?< DFK?<I D8P 8:HL@I< 8E 8CC<I>@:I<8:K@FE KF K?< 8EK@><E @= J?< ?8J 8EFK?<I :?@C; N@K? 9CFF; KPG<

  K?< :?@C; N@CC 9< @E ;8E><I LEC<JJ K?< DFK?<I K8B<J 8E 8CC<I>@:JLGGI<JJ8EK ;LI@E> GI<>E8E:P /?LJ G<FGC< C<8IE K?<@I 9CFF; KPG<J9<=FI<K?<PD8IIP

*? 8E; also: G<FGC< N@K? 9CFF; KPG< 8I< <8IE<JK 8E; :I<8K@M< N?@C< G<FGC< N@K? 9CFF; KPG< 8I< N@C; 8E; :?<<I=LC +<FGC< N@K?KPG< * 8I< 8>I<<89C< 8E; JF:@89C< N?@C< G<FGC< N@K? KPG< 8I<

:FFC 8E; :FEKIFCC<; h4FL NFLC; K?@EB K?8K * NFLC; 9< K?< 89J<E:<F= 8E; N?@C< NFLC; ALJK 9< GCLJ 9LK EF_i CC K?@J 8:e:FI;@E> KF K?< %8G8E<J< 9CFF; KPG< K?<FIP F= G<IJFE8C@KP  $K NFLC;J<<D K?8K 9CFF; KPG< GC8PJ K?< IFC< @E %8G8E K?8K 8JKIFCF>@:8C J@>EJGC8P @E K?< 2<JK I@>?K ;FNE KF 9CFF; KPG< ?FIFJ:FG<J @E K?< ;8@CP E<NJG8G<I

/?@J =8; @J <JG<:@8CCP F;; 9<:8LJ< 9CFF; KPG<J ?8M< never beenDPJK<I@FLJ EFK @E %8G8E 8E; EFK 8EPN?<I< 2< FECP BEFN 9CFF;KPG<J <M<E exist K?8EBJ KF &8IC '8E;JK<@E<I )F DPJK@: N@K:? ;F:eKFI EF M<E<I89C< JFI:<I<I <M<I J8@; 8 NFI; 89FLK 9CFF; KPG<J K?<I<8I< EF 8E:@<EK ;LJKP J:IFCCJ KF J?IFL; K?< <IIFI @E K?< 8LI8 F= 8EeK@HL@KP  $= K?< D<;@:8C GIF=<JJ@FE :C8@D<; KFDFIIFN K?8K @K ?8; 8CC9<<E 8 :FCFJJ8C ?F8O N< C8P=FCB NFLC; EFK ?8M< FE< J:I8G F= <M@e;<E:<=IFDFLILE8@;<;J<EJ<JKF:FEKI8;@:KK?<D

http://lesswrong.com/lw/nx/categorizing_has_consequences/  +8><b!8CC8:@<JF=FDGI<JJ@FEc

http://en.wikipedia.org/wiki/Japanese_blood_type_theory_of_personality +8><b"L8I;@8EJF=K?</ILK?c

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 258/309

/?<I<cJ E<M<I 9<<E 8 N8I 9<KN<<E 9CFF; KPG<J /?<I<cJ E<M<I<M<E 9<<E 8 GFC@K@:8C :FE=C@:K 9<KN<<E 9CFF; KPG<J /?< JK<I<FKPG<JDLJK?8M<8I@J<E strictly =IFDK?< mere existence F=K?<C89<CJ

)FN JFD<FE< @J 9FLE; KF GF@EK FLK K?8K K?@J @J 8 JKFIP F= :8Ke<>FI@Q@E> ?LD8EJ F<J K?< J8D< K?@E> ?8GG<E @= PFL :8K<>FI@Q<GC8EKJ FI IF:BJ FI F==@:< =LIE@KLI< $ :8EcK I<:8CC I<8;@E> 89FLKJL:? 8E <OG<I@D<EK 9LK F= :FLIJ< K?8K ;F<JEcK D<8E FE< ?8JEcK9<<E ;FE< h$c; <OG<:K K?< :?@<= ;@==@:LCKP F= ;F@E> JL:? 8E <OG<Ie@D<EK NFLC; 9< =@E;@E> 8 GIFKF:FC K?8K ;@;EcK D@JC<8; K?< JL9A<:KJ@EKF K?@EB@E> K?8K J@E:< K?< C89<C N8J >@M<E PFL @K DLJK 9< J@>eE@=@:8EK JFD<?FNi .F N?@C< $ ;FEcK D<8E KF LG;8K< FE @D8>@E8IP <M@;<E:< $ NFLC; GI<;@:K 8 GFJ@K@M< I<JLCK =FI K?< <OG<I@D<EK $

 NFLC; <OG<:K K?<D KF =@E; K?8K D<I< C89<C@E> ?8; GFN<I FM<I 8CCK?@E>J8KC<8JK@EK?<?LD8E@D8>@E8K@FE

4FL :8E J<< K?@J @E K<IDJ F= J@D@C8I@KP :CLJK<IJ FE:< PFL ;I8N 8 9FLE;8IP 8IFLE; 8 >IFLG K?< D@E; JK8IKJ KIP@E> KF ?8IM<JK J@D@C8Ie@K@<J =IFD K?< >IFLG E; LE=FIKLE8K<CP K?< ?LD8E G8KK<IEe;<K<:eKFIJ J<<D KF FG<I8K< @E JL:? FM<I;I@M< K?8K N< J<< G8KK<IEJ N?<K?<IK?<PcI< K?<I< FI EFK 8 N<8BCP E<>8K@M< :FII<C8K@FE :8E 9< D@JK8B<E=FI8JKIFE>GFJ@K@M<FE<N@K?89@KF=J<C<:K@M<D<DFIP

4FL :8E J<< K?@J @E K<IDJ F= E<LI8C 8C>FI@K?DJ :I<8K@E> 8 E8D<=FI 8 J<K F= K?@E>J @J C@B< 8CCF:8K@E> 8 JL9E<KNFIB KF =@E; G8KK<IEJ @EK?<D

4FL :8E J<< K?@J @E K<IDJ F= 8 :FDGI<JJ@FE =8CC8:P  K?@E>J >@M<EK?< J8D< E8D< <E; LG ;LDG<; @EKF K?< J8D< D<EK8C 9L:B<K 9CLIeI@E>K?<DKF><K?<I@EKFK?<J8D<GF@EKFEK?<D8G

*I PFL :8E J<< K?@J @E K<IDJ F= K?< 9FLE;C<JJ ?LD8E 89@C@KP KFD8B< JKL== LG FLK F= K?@E 8@I 8E; 9<C@<M< @K 9<:8LJ< EF FE< :8E GIFM<

@KcJ NIFE> 

J JFFE 8J PFL E8D< K?< :8K<>FIP PFL :8E JK8IK D8B@E> LG JKL== 89FLK @K /?< E8D<; K?@E> ;F<JEcK ?8M< KF 9< G<I:<GK@9C<@K;F<JEcK?8M<KF<O@JK@K;F<JEcK<M<E?8M<KF9<:F?<I<EK

 E; EF @KcJ EFK ALJK %8G8E #<I< @E K?< 2<JK 8 9CFF;eKPG<e98J<;;@<K9FFB:8CC<; 8K-@>?K4FLI/PG< N8J89<JKJ<CC<I

http://lesswrong.com/lw/kj/no_one_knows_what_science_doesnt_know/  +8><b.@D@C8I@KPCLJK<IJc +8><b)<LI8C8K<>FI@<Jc +8><b!8CC8:@<JF=FDGI<JJ@FEc

+8><b&EFN@E>9FLK@8J<J8E#LIK+<FGC<c http://en.wikipedia.org/wiki/Blood_type_diet 

-%&6+-!,'/'*+

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 259/309

 EP N8P PFL CFFB 8K @K ;I8N@E> 8 9FLE;8IP @E K?@E>JG8:< @J EFK8 E<LKI8C 8:K (8P9< 8 DFI< :C<8ECP ;<J@>E<; DFI< GLI<CP 8P<J@8E

 $ :FLC; GFE;<I 8E 8I9@KI8IP :C8JJ 8E; EFK 9< @E=CL<E:<; 9P @K LK

 PFL 8 ?LD8E ;F EFK ?8M< K?8K FGK@FE 8K<>FI@<J 8I< EFK JK8K@:K?@E>J @E K?< :FEK<OK F= 8 ?LD8E 9I8@E 8J JFFE 8J PFL 8:KL8CCP K?@EBF= K?<D K?<P <O<IK =FI:< FE PFLI D@E; *E< DFI< I<8JFE EFK KF9<C@<M<PFL:8E;<=@E<8NFI;8EPN8PPFLC@B<

+8><b/?<CLJK<I.KIL:KLI<F=/?@E>JG8:<c

,'*!2!&+'&+)-&+

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 260/309

19. Sneaking in Connotations

Followup to 8K<>FI@Q@E>#8JFEJ<HL<E:<J

4<JK<I;8P N< J8N K?8K @E %8G8E 9CFF; KPG<J ?8M< K8B<E K?<GC8:< F= 8JKIFCF>Pg@= PFLI 9CFF; KPG< @J =FI <O8DGC< PFLcI<JLGGFJ<;KF9<`:FFC8E;:FEKIFCC<;a

.F JLGGFJ< N< ;<:@;<; KF @EM<EK 8 E<N NFI; `N@>>@Ea 8E; de- fined K?@JNFI;KFD<8EG<FGC<N@K?>I<<E<P<J8E;9C8:B?8@Ig 

 >I<<Ee<P<;D8EN@K?9C8:B?8@IN8CB<;@EKF8 I<JK8LI8EK

a#8aJ8@;8EEPN8K:?@E>=IFD8E<8I9PK89C<`;@; PFLJ<<K?8K N@>>@EALJKN8CB<;@EKFK?<IFFDCFF;PN@>>@EJ FDD@K8CCJFIKJF=:I@D<JK?<P;Fa

#@JJ@JK<II;8J@>?<; `4FL?8M<EcK seen ?@D:FDD@K8EP:I@D<J?8M<PFL8EEPa

aFEcKE<<;KFa8EEPJ8@;GIF;L:@E>8;@:K@FE8IP`.<<@KJ8PJI@>?K?<I<@EK?<*O=FI;E>C@J?@:K@FE8IPb2@>>@E hi G<IJFEN@K?>I<<E<P<J8E;9C8:B?8@Ic#<cJ>FK>I<<E<P<J8E;9C8:B?8@I?<cJ8N@>>@E 4FLcI<

EFK>F@E>KF8I>L<N@K?K?<*O=FI;E>C@J?@:K@FE8IP8I<PFL By definition, 8>I<<Ee<P<;9C8:Be?8@I<;G<IJFE@J8N@>>@E.” 

aLKPFL:8CC<;?@D8N@>>@EaJ8@;I;8 `/?8KcJ8 E8JKPK?@E>KFJ8P89FLKJFD<FE<PFL;FEcK<M<EBEFN4FLcM<>FKEF<M@;<E:<K?8K?<GLKJKFFDL:?B<K:?LGFE?@J9LI><IJFIK?8K8J8B@;?<LJ<;?@JJC@E>J?FKKFC8LE:?989PJHL@II<CJa

`LK?< is 8N@>>@Ea8EEPJ8@;G8K@<EKCP `#<cJ>FK

 >I<<E<P<J8E;9C8:B?8@II@>?K %LJKPFLN8K:?8JJFFE8J?@J9LI><I8II@M<J?<cJI<8:?@E>=FIK?<B<K:?LGa

/?< ?LD8E D@E; G8JJ<J =IFD F9J<IM<; :?8I8:K<I@JK@:J KF @Ee=<II<; :?8I8:K<I@JK@:J M@8 K?< D<;@LD F= NFI;J $E `CC ?LD8EJ 8I<DFIK8C .F:I8K<J @J 8 ?LD8E K?<I<=FI< .F:I8K<J @J DFIK8C` K?< F9eJ<IM<; :?8I8:K<I@JK@:J 8I< .F:I8K<JcJ :CFK?<J JG<<:? KFFC LJ< 8E;

http://lesswrong.com/lw/ny/sneaking_in_connotations/ 

+8><b8K<>FI@Q@E>#8JFEJ<HL<E:<Jc +8><b/?<+8I89C<F=#<DCF:Bc

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 261/309

 ><E<I8CCP ?LD8E J?8G< K?< :8K<>FI@Q8K@FE @J `?LD8Ea K?< @E=<II<;:?8I8:K<I@JK@:@JGF@JFE89@C@KP9P?<DCF:B

*= :FLIJ< K?<I<cJ EF ?8I; ;@JK@E:K@FE 9<KN<<E `F9J<IM<; :?8I8:e

K<I@JK@:Ja 8E; @E=<II<; :?8I8:K<I@JK@:Ja $= PFL ?<8I JFD<FE< JG<8BK?<PcI< GIF989CP J?8G<; C@B< 8 ?LD8E 8CC <CJ< 9<@E> <HL8C $= PFLJ<< 8 ?LD8E =@>LI< @E K?< J?8;FNJ K?<E :<K<I@J G8I@9LJ @K :8E GIF9e89CPJG<8B

 E; P<K JFD< GIFG<IK@<J ;F K<E; KF 9< DFI< @E=<II<; K?8E F9eJ<IM<; 4FLcI< DFI< C@B<CP KF ;<:@;< K?8K JFD<FE< @J ?LD8E 8E;

 N@CC K?<I<=FI< 9LIE @= <OGFJ<; KF FG<E =C8D< K?8E :8IIP K?IFL>? K?<@E=<I<E:<K?<FK?<IN8P8IFLE;

$= PFL CFFB @E 8 ;@:K@FE8IP =FI K?< ;<=@E@K@FE F= `?LD8Ea PFLcI<DFI< C@B<CP KF =@E; :?8I8:K<I@JK@:J C@B< `@EK<CC@><E:<a 8E; `=<8K?<IeC<JJ 9@G<;ag:?8I8:K<I@JK@:J K?8K 8I< LJ<=LC =FI HL@:BCP <P<98CC@E> 

 N?8K @J 8E; @JEcK 8 ?LD8EgI8K?<I K?8E K?< K<E K?FLJ8E; :FEEFeK8K@FEJ =IFD MLCE<I89@C@KP KF ?<DCF:B KF FM<I:FE=@;<E:< K?8K N<:8E @E=<I =IFD JFD<FE<cJ 9<@E> ?LD8E 2?P +<I?8GJ ;@:K@FE8Ie@<J 8I< @EK<E;<; KF C<K PFL D8K:? LG C89<CJ KF J@D@C8I@KP >IFLGJ 8E;JF 8I< ;<J@>E<; KF HL@:BCP @JFC8K< :CLJK<IJ @E K?@E>JG8:< *I G<Ie?8GJ K?< 9@> ;@JK@E>L@J?@E> :?8I8:K<I@JK@:J 8I< K?< DFJK J8C@<EK 8E;

K?<I<=FI< =@IJK KF GFG @EKF 8 ;@:K@FE8IP <;@KFIcJ D@E; h$cD EFK JLI<?FN8N8I<;@:K@FE8IP<;@KFIJ8I<F=N?8KK?<P really ;Fi

LK K?< LGJ?FK @J K?8K N?<E 8EEP GLCCJ FLK ?@J * KF CFFBLG `N@>>@Ea ?< J<<J C@JK<; FECP K?< =@IJKe>C8E:< :?8I8:K<I@JK@:J K?8K;@JK@E>L@J? 8 N@>>@E "I<<E <P<J 8E; 9C8:B ?8@I /?< * ;F<JEcKC@JK K?< D8EP D@EFI connotations K?8K ?8M< :FD< KF 8KK8:? KF K?@JK<ID JL:? 8J :I@D@E8C GIF:C@M@K@<J :LC@E8IP G<:LC@8I@K@<J 8E; JFD<LE=FIKLE8K<:?@C;?FF;8:K@M@K@<J

#FN ;@; K?FJ< :FEEFK8K@FEJ ><K K?<I< @E K?< =@IJK GC8:< (8P9<

K?<I< N8J FE:< 8 =8DFLJ N@>>@E N@K? K?FJ< GIFG<IK@<J *I D8P9<JFD<FE< D8;< JKL== LG 8K I8E;FD 8E; NIFK< 8 J<I@<J F= 9<JKJ<CC@E> 9FFBJ 89FLK @K h The Wiggin, Talking to Wiggins, Raising Your LittleWiggin, Wiggins in the Bedroom i (8P9< <M<E K?< N@>>@EJ 9<C@<M< @KEFN 8E; 8:K 8::FI;@E>CP J JFFE 8J PFL :8CC JFD< G<FGC< `N@>e

 >@EJaK?<NFI;N@CC9<>@E8:HL@I@E>:FEEFK8K@FEJ

LK I<D<D9<I K?< +8I89C< F= #<DCF:B $= N< >F 9P K?< CF>@:8C:C8JJ ;<=@E@K@FEJ N< :8E E<M<I :C8JJ .F:I8K<J 8J 8 `?LD8Ea LEK@C

+8><b/?<+8I89C<F=#<DCF:Bc

+ &#!&!&'&&',,!'&+

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 262/309

8=K<I N< F9J<IM< ?@D KF 9< DFIK8C 2?<E<M<I JFD<FE< GLCCJ 8 ;@:eK@FE8IP K?<PcI< ><E<I8CCP KIP@E> KF JE<8B @E 8  connotation, EFK K?<8:KL8C;<=@E@K@FENI@KK<E;FNE@EK?<;@:K@FE8IP

 =K<I 8CC @= K?< only D<8E@E> F= K?< NFI; `N@>>@Ea @J `>I<<Ee<P<;9C8:Be?8@I<; G<IJFEa K?<E N?P EFK ALJK :8CC K?FJ< G<FGC< `>I<<Ee<P<; 9C8:Be?8@I<; G<FGC<a E; @= PFLcI< NFE;<I@E> N?<K?<I JFD<eFE< @J 8 B<K:?LGeI<8:?<I N?P EFK 8JB ;@I<:KCP  `$J ?< 8 B<K:?LGeI<8:?<Ia I8K?<I K?8E `$J ?< 8 N@>>@Ea h)FK< JL9JK@KLK@FE F= JL9eJK8E:<=FIJPD9FCi

*? 9LK 8I>L@E> K?< real  HL<JK@FE NFLC; I<HL@I< work. 4FLc;?8M< KF 8:KL8CCP N8K:? K?< N@>>@E KF J<< @= ?< I<8:?<; =FI K?<B<K:?LG *I D8P9< J<< @= PFL :8E =@E; JK8K@JK@:J FE ?FN D8EP 

 >I<<Ee<P<; 9C8:Be?8@I<; G<FGC< 8:KL8CCP C@B< B<K:?LG K 8EP I8K< PFL NFLC;EcK 9< 89C< KF ;F @K J@KK@E> @E PFLI C@M@E> IFFD N@K? PFLI<P<J :CFJ<; E; G<FGC< 8I< C8QP /?<Pc; I8K?<I 8I>L< `9P ;<=@E@eK@FEa <JG<:@8CCP J@E:< K?<P K?@EB `PFL :8E ;<=@E< 8 NFI; 8EP N8P 

 PFLC@B<a

LK F= :FLIJ< K?< real  I<8JFE K?<P :8I< N?<K?<I JFD<FE< @J8 `N@>>@Ea @J 8 :FEEFK8K@FEg8 =<<C@E> K?8K :FD<J 8CFE> N@K? K?<

 NFI;gK?8K@JEcK@EK?<;<=@E@K@FEK?<P claim KFLJ<

$D8>@E< 8EEP J8P@E> `'FFB ?<cJ >FK >I<<E <P<J 8E; 9C8:B?8@I #<cJ 8 N@>>@E $K J8PJ JF I@>?K K?<I< @E K?< ;@:K@FE8IPg there- fore, ?<cJ>FK9C8:B?8@I I>L<N@K?K?8K@=PFL:8Ea

F<JEcK ?8M< DL:? F= 8 KI@LDG?8EK I@E> KF @K ;F<J @K $= K?<I<8C GF@EK F= K?< 8I>LD<EK 8:KL8CCP was :FEK8@E<; @E K?< ;@:K@FE8IP ;<=@E@K@FEg@= K?< 8I>LD<EK ><EL@E<CP was CF>@:8CCP M8C@;gK?<E K?<8I>LD<EK NFLC; feel <DGKP @K NFLC; <@K?<I J8P EFK?@E> E<N FI 9<> K?<HL<JK@FE

$KcJ FECP K?< 8KK<DGK KF JDL>>C< @E :FEEFK8K@FEJ not <OGC@:@KCP C@JK<; @E K?< ;<=@E@K@FE K?8K D8B<J 8EPFE< =<<C K?<P :8E score a point K?8KN8P

+8><b2FI;J8J#@;;<E$E=<I<E:<Jc

+8><b#L>K?<,L<IPc +8><b-<GC8:<K?<.PD9FCN@K?K?<.L9JK8E:<c

-%&6+-!,'/'*+

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 263/309

20. Arguing “By Definition”

Followup to .E<8B@E>@EFEEFK8K@FEJ

`/?@J GCL:B<; :?@:B<E ?8J KNF C<>J 8E; EF =<8K?<IJgK?<I<=FI<by definition, @K@J8?LD8Ea

2?<E G<FGC< 8I>L< ;<=@E@K@FEJ K?<P LJL8CCP JK8IK N@K? JFD< M@Je@9C< BEFNE FI 8K C<8JK N@;<CP 9<C@<M<; J<K F= :?8I8:K<I@JK@:J K?<EGLCC FLK 8 ;@:K@FE8IP 8E; GF@EK FLK K?8K K?<J< :?8I8:K<I@JK@:J =@KK?< ;@:K@FE8IP ;<=@E@K@FE 8E; JF :FE:CL;< `/?<I<=FI< by definition,8K?<@JD@J8I<C@>@FEa

LK M@J@9C< BEFNE N@;<CP 9<C@<M<; :?8I8:K<I@JK@:J 8I< I8I<CP 

K?< I<8C GF@EK F= 8 ;@JGLK< %LJK K?< =8:K K?8K JFD<FE< [email protected]:I8K<JcJ KNF C<>J 8I< <M@;<EK <EFL>? KF D8B< 8 >FF; GI<D@J< =FIK?< 8I>LD<EK `/?<I<=FI< by definition, .F:I8K<J @J ?LD8Ea @E;@e:8K<J K?8K 9@G<;8C@JD GIF989CP @JEcK really N?8KcJ 8K JK8B<gFI K?<C@JK<E<I NFLC; I<GCP `2?8;;8P8 D<8E .F:I8K<J @J 9@G<;8C /?8KcJ

 N?8KN<cI<8I>L@E>89FLK@EK?<=@IJKGC8:<a

)FN K?<I< @J 8E @DGFIK8EK J<EJ< @E N?@:? N< :8E C<>@K@D8K<CP DFM< =IFD <M@;<EK :?8I8:K<I@JK@:J KF EFKeJFe<M@;<EK FE<J 4FL :8EC<>@K@D8K<CP J<< K?8K .F:I8K<J @J ?LD8EeJ?8G<; 8E; GI<;@:K ?@J MLCeE<I89@C@KP KF ?<DCF:B LK K?@J probabilistic @E=<I<E:< ;F<J EFK I<CP FE ;@:K@FE8IP ;<=@E@K@FEJ FI :FDDFE LJ8>< @K I<C@<J FE K?< LE@M<IJ<:FEK8@E@E> <DG@I@:8C:CLJK<IJ F= J@D@C8IK?@E>J

/?@J :CLJK<I JKIL:KLI< @J EFK >F@E> KF :?8E>< ;<G<E;@E> FE ?FN  PFL ;<=@E< PFLI NFI;J M<E @= PFL CFFB LG K?< ;@:K@FE8IP ;<=@E@eK@FE F= `?LD8Ea 8E; @K J8PJ `8CC =<8K?<IC<JJ 9@G<;J <O:<GK .F:I8K<JaK?8K @JEcK >F@E> KF :?8E>< K?< actual ;<>I<< KF N?@:? .F:I8K<J @J J@De@C8IKFK?<I<JKF=LJ=<8K?<IC<JJ9@G<;J

2?<E PFL 8I< 8I>L@E> correctly =IFD :CLJK<I JKIL:KLI< PFLcCC J8P JFD<K?@E> C@B< `.F:I8K<J ?8J KNF 8IDJ KNF =<<K 8 EFJ< 8E; KFE>L<JG<8BJ =CL<EK "I<<B LJ<J KFFCJ 8E; @E <M<IP 8JG<:K $cM< 9<<E 89C<KF F9J<IM< ?@D J<<DJ KF ?8M< <M<IP D8AFI 8E; D@EFI GIFG<IKP K?8K:?8I8:K<I@Q<J Homo sapiens; JF $cD >F@E> KF >L<JJ K?8K ?< ?8J ?LD8E

http://lesswrong.com/lw/nz/arguing_by_definition/  +8><b.E<8B@E>@EFEEFK8K@FEJc +8><b@J>L@J<;,L<I@<Jc +8><b.E<8B@E>@EFEEFK8K@FEJc

+8><b/?<CLJK<I.KIL:KLI<F=/?@E>JG8:<c +8><b.@D@C8I@KPCLJK<IJc

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 264/309

) ?LD8E 9@F:?<D@JKIP 8E; @J MLCE<I89C< KF ?<DCF:B ALJK C@B<8CC FK?<I Homo sapiens @E N?FD ?<DCF:B ?8J 9<<E :C@E@:8CCP K<JK<;=FIC<K?8C@KPa

 E; JLGGFJ< $ I<GCP `LK $ J8N .F:I8K<J FLK @E K?< =@<C;J N@K?JFD< ?<I9FCF>@JKJ $ K?@EB K?<P N<I< KIP@E> KF GI<G8I< 8E 8EK@;FK</?<I<=FI< $ don’t  <OG<:K .F:I8K<J KF B<<C FM<I 8=K<I ?< ;I@EBJ K?<?<DCF:Bg?< N@CC 9< 8E <O:<GK@FE KF K?< ><E<I8C 9<?8M@FI F= F9A<:KJ@E?@J:CLJK<IK?<P;@;EFKK8B<8E8EK@;FK<8E;?<;@;a

)FN K?<I<cJ EFK DL:? GF@EK @E 8I>L@E> FM<I N?<K?<I .F:I8K<J @J`?LD8Ea FI EFK /?< :FEM<IJ8K@FE ?8J KF DFM< KF 8 DFI< ;<K8@C<;C<M<C GFB< 8IFLE; inside K?< ;<K8@CJ K?8K D8B< LG K?< `?LD8Ea :8Ke<>FIPgK8CB 89FLK ?LD8E 9@F:?<D@JKIP 8E; JG<:@=@:8CCP K?< E<LIFeKFO@:<==<:KJF=:FE@@E<

$= PFL >F FE @EJ@JK@E> `LK .F:I8K<J @J 8 ?LD8E 8E; ?LD8EJ by definition, 8I< DFIK8Ca K?<E N?8K PFLcI< I<8CCP KIP@E> KF ;F @J 9CLIFLK <M<IPK?@E> PFL BEFN 89FLK .F:I8K<J except  K?< =8:K F= ?@J ?LeD8E@KPg@EJ@JK K?8K K?< FECP :FII<:K GI<;@:K@FE @J K?< FE< PFL NFLC;D8B< @= PFL BE<N EFK?@E> 89FLK .F:I8K<J except K?8K ?< N8J ?LD8E

2?@:? @J C@B< @EJ@JK@E> K?8K 8 :F@E @J n C@B<CP KF 9< J?FN@E> ?<8;J FI K8@CJ 9<:8LJ< @K @J 8 `=8@I :F@Ea 8=K<I PFLcM< actually looked at 

the coin 8E; @KcJ J?FN@E> ?<8;J $KcJ C@B< @EJ@JK@E> K?8K !IF;F ?8J K<E=@E><IJ 9<:8LJ< DFJK ?F99@KJ ?8M< K<E =@E><IJ 8=K<I PFLcM< already looked at his hands 8E; J<<E E@E< =@E><IJ )8KLI8CCP K?@J @J @CC<>8C LEe;<I 8P<J@8E GIF989@C@KP K?<FIP 4FL :8EcK ALJK I<=LJ< KF :FE;@K@FEFEE<N<M@;<E:<

 E; PFL :8EcK ALJK B<<G FE< :8K<>FI@Q8K@FE 8E; D8B< <JK@D8K<J98J<; FE K?8K N?@C< ;<C@9<I8K<CP K?IFN@E> FLK <M<IPK?@E> <CJ< PFLBEFN

)FK <M<IP G@<:< F= E<N <M@;<E:< D8B<J 8 J@>E@=@:8EK ;@==<I<E:<F= :FLIJ< $= $ J<< K?8K .F:I8K<J ?8J E@E< =@E><IJ K?@J @JEcK >F@E> KFEFK@:<89CP :?8E>< DP <JK@D8K< F= ?@J MLCE<I89@C@KP KF ?<DCF:B 9<e:8LJ< $cCC <OG<:K K?8K K?< N8P .F:I8K<J CFJK ?@J =@E><I ;@;EcK :?8E><K?< I<JK F= ?@J 9@F:?<D@JKIP E; K?@J @J KIL< whether or not K?< ;@:eK@FE8IPcJ ;<=@E@K@FE J8PJ K?8K ?LD8E 9<@E>J ?8M< K<E =@E><IJ /?<C<>8C @E=<I<E:< @J 98J<; FE K?< :CLJK<I JKIL:KLI< F= K?< <EM@IFED<EK8E; K?< :8LJ8C JKIL:KLI< F= 9@FCF>P not N?8K K?< ;@:K@FE8IP <;@KFI

 NI@K<J;FNEEFI<M<E`:FDDFELJ8><a

+8><b-<GC8:<K?<.PD9FCN@K?K?<.L9JK8E:<c

-%&6+-!,'/'*+

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 265/309

)FN FI;@E8I@CP N?<E PFLcI< ;F@E> K?@J right  g@E 8  legitimate N8PgPFL ALJK J8P `/?< :FE@@E< 8CB8CF@; =FLE; @E ?<DCF:B GIFe;L:<J DLJ:LC8I G8I8CPJ@J @E ?LD8EJ I<JLCK@E> @E ;<8K? 9P 8JG?POe

@8K@FEa *I DFI< J@DGCP `#LD8EJ 8I< MLCE<I89C< KF ?<DCF:Ba/?8KcJ?FN@KcJLJL8CCPJ8@;@E8 legitimate 8I>LD<EK

2?<E NFLC; JFD<FE< =<<C K?< E<<; KF strengthen K?< 8I>LD<EK N@K? K?< <DG?8K@: G?I8J< `9P ;<=@E@K@FEa h$< `#LD8EJ 8I< MLCeE<I89C< KF ?<DCF:B by definition!  i 2?P N?<E K?< @E=<II<; :?8I8:eK<I@JK@: ?8J 9<<E :8CC<; @EKF ;FL9Kg.F:I8K<J ?8J 9<<E J<<E :FEJLCKe@E> ?<I9FCF>@JKJg8E; JF K?< JG<8B<I =<<CJ K?< E<<; KF K@>?K<E K?<

 M@J<F=CF>@:

.F N?<E PFL J<< `9P ;<=@E@K@FEa LJ<; C@B< K?@J @K LJL8CCP D<8EJ

`!FI><K N?8K PFLcM< ?<8I; 89FLK .F:I8K<J :FEJLCK@E> ?<I9FCFe >@JKJg?LD8EJ by definition, 8I<DFIK8Ca

+<FGC< =<<C K?< E<<; KF JHL<<Q< K?< 8I>LD<EK FEKF 8 J@E>C<:FLIJ< 9P J8P@E> `EP + 9P ;<=@E@K@FE ?8J GIFG<IKP ,a FE <O8:KCP K?FJ< F::8J@FEJ N?<E K?<P J<< 8E; GI<=<I KF ;@JD@JJ FLK F= ?8E; additional arguments K?8K :8CC @EKF ;FL9K K?< ;<=8LCK @E=<I<E:< 98J<;FE:CLJK<I@E>

.F KFF N@K? K?< 8I>LD<EK `3 by definition, @J 8 4a > `K?<e

@JKJ 9<C@<M< K?8K "F; ;F<JEcK <O@JK K?<I<=FI< 8K?<@JKJ ?8M< 9<C@<=J89FLK "F; 9<:8LJ< 8 E<>8K@M< 9<C@<= @J JK@CC 8 9<C@<= K?<I<=FI< 8K?<e@JD 8JJ<IKJ 8EJN<IJ KF K?<FCF>@:8C HL<JK@FEJ K?<I<=FI< 8K?<@JD @Jby definition, 8I<C@>@FEa

4FL NFLC;EcK =<<C K?< E<<; KF J8P `#@E;L@JD by definition, @J 8 I<C@>@FEa 9<:8LJ< N<CC F= :FLIJ< #@E;L@JD @J 8 I<C@>@FE $KcJ EFKALJK8I<C@>@FE`9P;<=@E@K@FEa@KcJC@B<8E actual I<C@>@FE

 K?<@JD ;F<J EFK I<J<D9C< K?< :<EKI8C D<D9<IJ F= K?< `I<C@e >@FEa :CLJK<I JF @= @K N8JEcK =FI K?< =8:K K?8K 8K?<@JD @J 8 I<C@>@FE by definition, PFL D@>?K >F 8IFLE; K?@EB@E> K?8K 8K?<@JD wasn’t  8 I<C@e >@FE /?8KcJ N?P PFLcM< >FK KF :ILJ? 8CC FGGFJ@K@FE 9P GF@EK@E> FLKK?8K `K?<@JD @J 8 I<C@>@FEa @J KIL< by definition 9<:8LJ< @K @JEcK KIL<8EPFK?<IN8P

2?@:? @J KF J8P +<FGC< @EJ@JK K?8K `3 by definition, @J 8 4a FEK?FJ< F::8J@FEJ N?<E K?<PcI< KIP@E> KF JE<8B @E 8 :FEEFK8K@FE F= 4K?8K @JEcK ;@I<:KCP @E K?< ;<=@E@K@FE 8E; 3 ;F<JEcK CFFB 8CC K?8K DL:?C@B<FK?<ID<D9<IJF=K?<4:CLJK<I

+8><b.E<8B@E>@EFEEFK8K@FEJc

*-!&41!&!,!'&5

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 266/309

*M<I K?< C8JK K?@IK<<E P<8IJ $cM< 9<<E B<<G@E> KI8:B F= ?FN F=K<E K?@J G?I8J< @J LJ<; :FII<:KCP M<IJLJ @E:FII<:KCPgK?FL>? EFK

 N@K? C@K<I8C JK8K@JK@:J $ =<8I LK <P<98CC@E> JL>><JKJ K?8K LJ@E> K?<

G?I8J< by definition, 8EPN?<I< FLKJ@;< F= D8K? @J 8DFE> K?< DFJK8C8ID@E> J@>E8CJ F= =C8N<; 8I>LD<EK $cM< <M<I =FLE; $KcJ I@>?K LGK?<I< N@K? `#@KC<I` `"F;a `89JFCLK<CP :<IK8@Ea 8E; `:8EcK GIFM<K?8Ka

/?@J ?<LI@JK@: F= =8@CLI< @J EFK G<I=<:KgK?< =@IJK K@D< $ <M<IJGFKK<; 8 :FII<:K LJ8>< FLKJ@;< F= D8K? @K N8J 9P -@:?8I; !<PEeD8E 8E; J@E:< K?<E $cM< JGFKK<; DFI< LK PFLcI< GIF989CP 9<KK<IF== ALJK ;<C<K@E> K?< G?I8J< `9P ;<=@E@K@FEa =IFD PFLI MF:89LeC8IPg8E; always FE 8EP F::8J@FE N?<I< PFL D@>?K 9< K<DGK<; KF

J8P @K @E @K8C@:J FI =FCCFN<; N@K? 8E <O:C8D8K@FE D8IB /?8KcJ 8 98;@;<8 by definition! 

http://www.catb.org/jargon/html/G/Godwins-Law.html 

-%&6+-!,'/'*+

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 267/309

21. Where to Draw the Boundary?

Followup to I>L@E>`P<=@E@K@FEa

/?<FE<:FD<JKFPFL8E; J8PJ

'FE>?8M<$GFE;<I<;K?<D<8E@E>F=K?<NFI;`IKa8E;8KC8JK$cM<=FLE;N?8KJ<<DJKFD<8J8K@J=8:KFIP ;<=@E@K@FE`IK@JK?8KN?@:?@J;<J@>E<;=FIK?<GLIGFJ<F=:I<8K@E>8I<8:K@FE@E8E8L;@<E:<a

 Just because there’s a word “art” doesn’t mean that it has a meaning, floating out there in the void, which you can discover by finding the right  definition.

$K =<<CJK?8KN8P 9LK@K@JEFKJF

2FE;<I@E> ?FN KF define a word  D<8EJ PFLcI< CFFB@E> 8K K?<GIF9C<D K?< NIFE> N8PgJ<8I:?@E> =FI K?< DPJK<I@FLJ <JJ<E:< F= 

 N?8K@J@E=8:K8 :FDDLE@:8K@FEJ@>E8C

)FN K?<I< is 8 I<8C :?8CC<E>< N?@:? 8 I8K@FE8C@JK D8P C<>@K@D8K<eCP 8KK8:B 9LK K?< :?8CC<E>< @J EFK KF =@E; 8 J8K@J=8:KFIP ;<=@E@K@FEF= 8 NFI; /?< I<8C :?8CC<E>< :8E 9< GC8P<; 8J 8 J@E>C<eGC8P<I >8D<

 N@K?FLK JG<8B@E> 8CFL; /?< :?8CC<E>< @J =@>LI@E> FLK N?@:? K?@E>J8I< J@D@C8I KF <8:? FK?<IgN?@:? K?@E>J 8I< :CLJK<I<; KF><K?<Ig8E;JFD<K@D<JN?@:?K?@E>J?8M<8:FDDFE:8LJ<

$= PFL ;<=@E< `<CL:KIFDL>E<K@JDa KF @E:CL;< C@>?KE@E> @E:CL;<:FDG8JJ<J <O:CL;< C@>?K 8E; @E:CL;< (<JD<IcJ `8E@D8C D8>eE<K@JDa hN?8K N< EFN :8CC ?PGEFJ@Ji K?<E PFL N@CC ?8M< JFD< KIFLe9C< 8JB@E> `#FN ;F<J <C<:KIFDL>E<K@JD NFIBa 4FL ?8M< CLDG<;KF><K?<I K?@E>J N?@:? ;F EFK 9<CFE> KF><K?<I 8E; <O:CL;<; FK?<IJ

K?8K NFLC; 9< E<<;<; KF :FDGC<K< 8 J<K h/?@J <O8DGC< @J ?@JKFI@e:8CCPGC8LJ@9C<(<JD<I:8D<9<=FI<!8I8;8Pi

2< :FLC; J8P K?8K <C<:KIFDL>E<K@JD @J 8  wrong word, 8 9FLE;e8IP @E K?@E>JG8:< K?8K CFFGJ 8IFLE; 8E; JN<IM<J K?IFL>? K?< :CLJeK<IJ8:LKK?8K=8@CJKF:8IM<I<8C@KP8CFE>@KJE8KLI8CAF@EKJ

http://lesswrong.com/lw/o0/where_to_draw_the_boundary/  +8><bI>L@E>`P<=@E@K@FEac http://lesswrong.com/lw/nw/fallacies_of_compression/ia9 +8><b!<<CK?<(<8E@E>c

+8><b/?<I>LD<EK=IFDFDDFE0J8><c +8><b/?<CLJK<I.KIL:KLI<F=/?@E>JG8:<c

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 268/309

!@>LI@E> N?<I< KF :LK I<8C@KP @E FI;<I KF :8IM< 8CFE> K?<AF@EKJg this @J K?< GIF9C<D NFIK?P F= 8 I8K@FE8C@JK $K @J N?8K G<FGC< should 9< KIP@E> KF ;F N?<E K?<P J<K FLK @E J<8I:? F= K?< =CF8K@E> 

<JJ<E:<F=8NFI; E; D8B< EF D@JK8B< @K @J 8  scientific :?8CC<E>< KF I<8C@Q< K?8K

 PFL E<<; 8 J@E>C< NFI; KF ;<J:I@9< 9I<8K?@E> 8E; =@I< .F ;F EFKK?@EBKF:FEJLCKK?<;@:K@FE8IP<;@KFIJ=FIK?8K@JEFKK?<@IAF9

2?8K @J `8IKa LK K?<I< @J EF <JJ<E:< F= K?< NFI; =CF8K@E> @EK?<MF@;

+<I?8GJ PFL :FD< KF D< N@K? 8 CFE> C@JK F= K?< K?@E>J K?8K PFL:8CC`8IKa8E;`EFK8IKa

/?< Little Fugue in G Minor: IK GLE:?@EK?<EFJ< )FK8IKJ:?<IcJ Relativity: IK

 =CFN<I )FK8IK/?<+PK?FEGIF>I8DD@E>C8E>L8>< IK

 :IFJJ=CF8K@E>@ELI@E< )FK8IK %8:B18E:<cJ Tschai EFM<CJ IK(F;<IEIK )FK8IK

 E; PFL J8P KF D< `$K =<<CJ @EKL@K@M< KF D< KF ;I8N K?@J 9FLE;e8IP 9LK $ ;FEcK BEFN N?Pg:8E PFL =@E; D< 8E @EK<EJ@FE K?8KD8K:?<J K?@J <OK<EJ@FE 8E PFL >@M< D< 8  simple ;<J:I@GK@FE F= K?@J9FLE;8IPa

.F $ I<GCP `$ K?@EB @K ?8J KF ;F N@K? 8;D@I8K@FE F= :I8=KJD8EeJ?@G NFIB >F@E> @E 8E; NFE;<I :FD@E> FLK 2?8K K?< @E:CL;<;@K<DJ ?8M< @E :FDDFE @J K?< J@D@C8I 8<JK?<K@: <DFK@FEJ K?8K K?<P @EJG@I< 8E; K?< ;<C@9<I8K< ?LD8E <==FIK K?8K N<EK @EKF K?<D N@K?

K?<@EK<EKF=GIF;L:@E>JL:?8E<DFK@FEa$J K?@J ?<CG=LC FI @J @K ALJK :?<8K@E>  8K /89FF $ NFLC; 8I>L<

K?8K K?< C@JK F= N?@:? ?LD8E <DFK@FEJ 8I< FI 8I< EFK aesthetic @J =8IDFI< :FDG8:K K?8E K?< C@JK F= <M<IPK?@E> K?8K @J FI @JEcK 8IK 4FLD@>?K 9< 89C< KF J<< K?FJ< <DFK@FEJ C@>?K@E> LG 8E =(-$ J:8Eg$J8PK?@J9PN8PF=<DG?8J@Q@E>K?8K<DFK@FEJ8I<EFK<K?<I<8C

http://lesswrong.com/lw/hq/universal_fire/ 

+8><b-<GC8:<K?<.PD9FCN@K?K?<.L9JK8E:<c +8><b/89FF4FLI2FI;Jc

-%&6+-!,'/'*+

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 269/309

LK F= :FLIJ< DP ;<=@E@K@FE F= 8IK @J EFK K?< I<8C GF@EK /?< I<8CGF@EK @J K?8K PFL :FLC; N<CC ;@JGLK< <@K?<I K?< @EK<EJ@FE FI K?< <OeK<EJ@FE F=DP;<=@E@K@FE

4FL :FLC; J8P `<JK?<K@: <DFK@FE @J not N?8K K?<J< K?@E>J ?8M<@E :FDDFE N?8K K?<P ?8M< @E :FDDFE @J 8E @EK<EK KF @EJG@I< any :FDGC<O <DFK@FE =FI K?< J8B< F= @EJG@I@E> @Ka /?8K NFLC; 9<;@JGLK@E> DP @EK<EJ@FE DP 8KK<DGK KF ;I8N 8 :LIM< K?IFL>? K?<;8K8 GF@EKJ 4FL NFLC; J8P `4FLI <HL8K@FE D8P IFL>?CP =@K K?FJ<GF@EKJ9LK@K@JEFKK?<KIL<><E<I8K@E>;@JKI@9LK@FEa

*I PFL :FLC; ;@JGLK< DP <OK<EJ@FE 9P J8P@E> `.FD< F= K?<J<K?@E>J ;F 9<CFE> KF><K?<Ig$ :8E J<< N?8K PFLcI< ><KK@E> 8Kg9LKK?< +PK?FE C8E>L8>< J?FLC;EcK 9< FE K?< C@JK 8E; (F;<IE IKJ?FLC; 9<a h/?@J NFLC; D8IB PFL 8J 8 >LCC@9C< G?@C@JK@E< 9LK PFL:FLC; 8I>L< @Ki #<I< K?< GI<JLDGK@FE @J K?8K K?<I< @J @E;<<; 8ELE;<ICP@E> :LIM< K?8K ><E<I8K<J K?@J 8GG8I<EK C@JK F= J@D@C8I 8E; ;@JeJ@D@C8I K?@E>JgK?8K K?<I< @J 8 I?PD< 8E; I<8JFE even though you haven’t said yet where it comes from g9LK $ ?8M< LEN@KK@E>CP CFJK K?<I?PK?D8E;@E:CL;<;JFD<;8K8GF@EKJ=IFD8;@==<I<EK><E<I8KFI

'FE> 9<=FI< PFL know N?8K @K @J K?8K <C<:KI@:@KP 8E; D8>E<K@JD?8M< @E :FDDFE PFL D@>?K JK@CC JLJG<:Kg98J<; FE JLI=8:< 8GG<8Ie

8E:<JgK?8K`8E@D8CD8>E<K@JDa;F<JEFK9<CFE>FEK?<C@JK*E:< LGFE 8 K@D< @K N8J K?FL>?K K?8K K?< NFI; `=@J?a @E:CL;<;

;FCG?@EJ )FN PFL :FLC; GC8P K?< F?eJFe:C<M<I 8I>L<I 8E; J8P`/?< C@JK l.8CDFE >LGG@<J J?8IBJ ;FCG?@EJ KIFLKm @J ALJK 8 C@JKgPFL :8EcK J8P K?8K 8 C@JK @J wrong. $ :8E GIFM< @E J<K K?<FIP K?8KK?@J C@JK <O@JKJ .F DP ;<=@E@K@FE F= fish N?@:? @J J@DGCP K?@J <OK<EeJ@FE8CC@JK:8EEFKGFJJ@9CP9<bNIFE>c8JPFL:C8@Da

*I PFL :FLC; JKFG GC8P@E> E@KN@K >8D<J 8E; 8;D@K K?8K ;FCG?@EJ;FEcK9<CFE>FEK?<=@J?C@JK

4FL :FD< LG N@K? 8 C@JK F= K?@E>J K?8K feel  J@D@C8I 8E; K8B< 8  >L<JJ 8K N?P K?@J @J JF LK N?<E PFL =@E8CCP ;@J:FM<I N?8K K?<P re- ally ?8M< @E :FDDFE @K D8P KLIE FLK K?8K PFLI >L<JJ N8J NIFE> $KD8P<M<EKLIEFLKK?8KPFLIC@JKN8JNIFE>

4FL :8EEFK ?@;< 9<?@E; 8 :FD=FIK@E> J?@<C; F= :FII<:Ke9Pe;<=@eE@K@FE FK? <OK<EJ@FE8C ;<=@E@K@FEJ 8E; @EK<EJ@FE8C ;<=@E@K@FEJ :8E9<NIFE>:8E=8@CKF:8IM<I<8C@KP8KK?<AF@EKJ

+8><bOK<EJ@FEJ8E;$EK<EJ@FEJc

/*,'*/,'-&*1

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 270/309

8K<>FI@Q@E> @J 8 >L<JJ@E> <E;<8MFI @E N?@:? PFL :8E D8B< D@JeK8B<J JF @KcJ N@J< KF 9< 89C< KF 8;D@K =IFD 8 K?<FI<K@:8C JK8E;GF@EKK?8KPFLI;<=@E@K@FEe>L<JJ<J:8E9<`D@JK8B<Ea

-%&6+-!,'/'*+

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 271/309

22. Entropy, and Short Codes

Followup to 2?<I<KFI8NK?<FLE;8IP

.LGGFJ< PFL ?8M< 8 JPJK<D 3 K?8KcJ <HL8CCP C@B<CP KF 9< @E 8EP F= GFJJ@9C<JK8K<J

l33333333m

/?<I<cJ 8E <OKI8FI;@E8I@CP L9@HL@KFLJ HL8EK@KPg@E G?PJ@:JD8K?<D8K@:J 8E; <M<E 9@FCF>Pg:8CC<; entropy 8E; K?< <EKIFGP F= 3 @J 9@KJ /?@J D<8EJ K?8K FE 8M<I8>< N<cCC ?8M< KF 8JB P<JeFIeEF HL<JK@FEJ KF =@E; FLK 3cJ M8CL< !FI <O8DGC< JFD<FE< :FLC; K<CCLJ3cJM8CL<LJ@E>K?@J:F;<

3 3 3 3

3 3 3 3

.F @= $ 8JB<; `$J K?< =@IJK JPD9FC a 8E; ?<8I; `P<Ja K?<E 8JB<;`$J K?< J<:FE; JPD9FC a 8E; ?<8I; `EFa K?<E 8JB<; `$J K?< K?@I;JPD9FCa8E;?<8I;`EFa$NFLC;BEFNK?8K3N8J@EJK8K<

)FN JLGGFJ< K?8K K?< JPJK<D 4 ?8J =FLI GFJJ@9C< JK8K<J N@K?K?<=FCCFN@E>GIF989@C@K@<J

4hni

4hni

4hni

4hni

/?<E K?< <EKIFGP F= 4 NFLC; 9< 9@KJ D<8E@E> K?8K N< :8E=@E;FLK@KJM8CL<9P8JB@E>P<JeFIeEFHL<JK@FEJ

2?8K ;F<J @K D<8E KF K8CB 89FLK 8JB@E> FE< 8E; K?I<<e=FLIK?JF= 8 HL<JK@FE $D8>@E< K?8K N< ;<J@>E8K< K?< JK8K<J F= 4 LJ@E> K?<=FCCFN@E>:F;<

4 4 4 4

http://lesswrong.com/lw/o1/entropy_and_short_codes/  +8><b2?<I<KFI8NK?<FLE;8IPc

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 272/309

!@IJK PFL 8JB `$J K?< =@IJK JPD9FC a $= K?< 8EJN<I @J `P<Ja PFLcI< ;FE< 4 @J @E JK8K< /?@J ?8GG<EJ ?8C= K?< K@D< JF n F= K?<K@D<@KK8B<JP<JeFIeEFHL<JK@FEKF=@E;FLK4cJJK8K<

.LGGFJ< K?8K @EJK<8; K?< 8EJN<I @J `)Fa /?<E PFL 8JB `$J K?<J<:FE; JPD9FC a $= K?< 8EJN<I @J `P<Ja PFLcI< ;FE< 4 @J @E JK8K< 4 @J @E JK8K< N@K? GIF989@C@KP 8E; <8:? K@D< 4 @J @E JK8K<

 N< ;@J:FM<I K?@J =8:K LJ@E> KNF P<JeFIeEF HL<JK@FEJ JF n F= K?<K@D<@KK8B<JHL<JK@FEJKF;@J:FM<I4cJJK8K<

$= K?< 8EJN<I @J `)Fa KN@:< @E 8 IFN PFL 8JB `$J K?< K?@I; JPDe9FC a $= `P<Ja PFLcI< ;FE< 8E; 4 @J @E JK8K< @= `EFa PFLcI< ;FE<8E; 4 @J @E JK8K< /?< F= K?< K@D< K?8K 4 @J @E JK8K< @K K8B<JK?I<< HL<JK@FEJ 8E; K?< F= K?< K@D< K?8K 4 @J @E JK8K< @K K8B<J

K?I<<HL<JK@FEJ

hihihihi

/?< ><E<I8C =FIDLC8 =FI K?< <EKIFGP F= 8 JPJK<D . @J K?< JLDFM<I8CC.@F=eGh.@ iCF> hGh.@ ii

!FI <O8DGC< K?< CF> h98J< i F= @J e .F eh ei @JK?< :FEKI@9LK@FE F= JK8K< . KF K?< KFK8C <EKIFGP F= K?< K@D< N<?8M<KF8JBHL<JK@FEJ

4FL :8EcK 8CN8PJ ;<M@J< 8 G<I=<:K :F;< =FI 8 JPJK<D 9LK @= PFL?8M< KF K<CC JFD<FE< K?< JK8K< F= 8I9@KI8I@CP D8EP :FG@<J F= . @E8 J@E>C< D<JJ8>< PFL :8E ><K 8I9@KI8I@CP :CFJ< KF 8 G<I=<:K :F;<h"FF>C<`8I@K?D<K@::F;@E>a=FI8J@DGC<D<K?F;i

)FN PFL D@>?K 8JB `2?P EFK LJ< K?< :F;< =FI 4 @EJK<8;F= 2FLC;EcKK?8KC<KLJKI8EJD@KD<JJ8><JDFI<HL@:BCPa

LK @= PFL LJ< K?< :F;< =FI 4 K?<E N?<E JFD<FE< 8EJN<IJ`4<Ja KF K?< HL<JK@FE `$J K?< =@IJK JPD9FC a PFL NFEcK BEFN P<K

 N?<K?<I K?< JPJK<D JK8K< @J 4 hi FI 4 hi $E =8:K @= PFL :?8E><K?< :F;< K?@J N8P K?< N?FC< JPJK<D =8CCJ 8G8IKg9<:8LJ< @= PFL ?<8I`a PFL ;FEcK BEFN @= @K D<8EJ `4 =FCCFN<; 9P 4a FI `4 =FCeCFN<;9P4a

/?<DFI8C@JK?8K short words are a conserved resource.

/?< B<P KF :I<8K@E> 8 >FF; :F;<g8 :F;< K?8K KI8EJD@KJ D<Je

J8><J 8J :FDG8:KCP 8J GFJJ@9C<g@J KF I<J<IM< J?FIK NFI;J =FI K?@E>J

-%&6+-!,'/'*+

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 273/309

K?8K PFLcCC E<<; KF J8P =I<HL<EKCP 8E; LJ< CFE><I NFI;J =FI K?@E>JK?8KPFLNFEcKE<<;KFJ8P8JF=K<E

2?<E PFL K8B< K?@J 8IK KF @KJ C@D@K K?< C<E>K? F= K?< D<JJ8>< PFLE<<; KF ;<J:I@9< JFD<K?@E> :FII<JGFE;J <O8:KCP FI 8CDFJK <O8:KeCP KF @KJ GIF989@C@KP /?@J @J K?< (@E@DLD <J:I@GK@FE '<E>K? FI(@E@DLD(<JJ8><'<E>K?=FID8C@Q8K@FEF= *::8DcJ-8QFI

 E; JF <M<E K?< labels K?8K N< LJ< =FI NFI;J 8I< EFK HL@K< 8Ie9@KI8IP /?< JFLE;J K?8K N< 8KK8:? KF FLI :FE:<GKJ :8E 9< 9<KK<IFI NFIJ< N@J<I FI DFI< =FFC@J? M<E 8G8IK =IFD :FEJ@;<I8K@FEJ F= :FDDFELJ8><

$ J8P 8CC K?@J 9<:8LJ< K?< @;<8 K?8K `4FL :8E 3 8EP N8P PFL C@B<a@J 8 ?L>< F9JK8:C< KF C<8IE@E> ?FN KF 3 N@J<CP `$KcJ 8 =I<< :FLEeKIP $ ?8M< 8 I@>?K KF DP FNE FG@E@FEa F9JKIL:KJ K?< 8IK F= =@E;@E> KILK? `$ :8E ;<=@E< 8 NFI; 8EP N8P $ C@B<a F9JKIL:KJ K?< 8IK F= :8IMe@E> I<8C@KP 8K @KJ AF@EKJ E; <M<E K?< J<EJ@9C<eJFLE;@E> `/?< C89<CJ

 N< 8KK8:? KF NFI;J 8I< 8I9@KI8IPa F9JKIL:KJ 8N8I<E<JJ F= :FDG8:KeE<JJ +IFJF;P KFF =FI K?8K D8KK<Ig/FCB@<E FE:< F9J<IM<; N?8K 8 9<8LK@=LC JFLE; K?< G?I8J< `:<CC8I ;FFIa D8B<J K?8K @J K?< B@E; F= 8N8I<E<JJ@KK8B<JKFLJ<C8E>L8><C@B</FCB@<E

/?< C<E>K? F= NFI;J 8CJF GC8PJ 8 EFEKI@M@8C IFC< @E K?< :F>E@K@M<J:@<E:<F=C8E>L8><

FEJ@;<I K?< G?I8J<J `I<:C@E<Ia `:?8@Ia 8E; `=LIE@KLI<a -<e:C@E<I @J 8 DFI< JG<:@=@: :8K<>FIP K?8E :?8@I =LIE@KLI< @J 8 DFI<

 ><E<I8C :8K<>FIP K?8E :?8@I LK K?< M8JK D8AFI@KP F= :?8@IJ ?8M< 8 :FDDFE LJ<gPFL LJ< K?< J8D< JFIK F= DFKFI 8:K@FEJ KF J@K ;FNE@E K?<D 8E; PFL J@K ;FNE @E K?<D =FI K?< J8D< JFIK F= GLIGFJ< hKFK8B< PFLI N<@>?K F== PFLI =<<K N?@C< PFL <8K FI I<8; FI KPG< FII<JKi -<:C@E<IJ ;F EFK ;<G8IK =IFD K?@J K?<D< `!LIE@KLI<a FE K?<FK?<I ?8E; @E:CL;<J K?@E>J C@B< 9<;J 8E; K89C<J N?@:? ?8M< ;@==<Ie<EKLJ<J8E;:8CCLG;@==<I<EKDFKFI=LE:K@FEJ=IFD:?8@IJ

$E K?< K<ID@EFCF>P F= :F>E@K@M< GJP:?FCF>P `:?8@Ia @J 8 basic-lev-el category.

+<FGC< ?8M< 8 K<E;<E:P KF K8CB 8E; GI<JLD89CP K?@EB 8K K?< 98eJ@: C<M<C F= :8K<>FI@Q8K@FEgKF ;I8N K?< 9FLE;8IP 8IFLE; `:?8@IJa

+8><b*::8DcJ-8QFIc +8><b/?<I>LD<EK=IFDFDDFE0J8><c

http://www.overcomingbias.com/2006/12/you_are_never_e.html  +8><b2?<I<KFI8NK?<FLE;8IPc

 &,*'(1&+'*,'+

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 274/309

I8K?<I K?8E 8IFLE; K?< DFI< JG<:@=@: :8K<>FIP `I<:C@E<Ia FI K?<DFI< ><E<I8C :8K<>FIP `=LIE@KLI<a +<FGC< 8I< DFI< C@B<CP KF J8P `4FL :8E J@K @E K?8K :?8@Ia K?8E `4FL :8E J@K @E K?8K I<:C@E<Ia FI

`4FL:8EJ@K@EK?8K=LIE@KLI<a E; @K @J EF :F@E:@;<E:< K?8K K?< NFI; =FI `:?8@Ia :FEK8@EJ

=<N<I JPCC89C<J K?8E <@K?<I `I<:C@E<Ia FI `=LIE@KLI<a 8J@:eC<M<C :8Ke<>FI@<J @E ><E<I8C K<E; KF ?8M< J?FIK E8D<J 8E; EFLEJ N@K? J?FIKE8D<J K<E; KF I<=<I KF 98J@:eC<M<C :8K<>FI@<J )FK 8 G<I=<:K ILC< F= :FLIJ< 9LK 8 ;<=@E@K< K<E;<E:P !I<HL<EK LJ< >F<J 8CFE> N@K? J?FIK

 NFI;JJ?FIKNFI;J>F8CFE>N@K?=I<HL<EKLJ<

*I 8J FL>C8J #F=JK8;K<I GLK @K K?<I<cJ 8 I<8JFE N?P K?< Ee >C@J? C8E>L8>< LJ<J K?<a KF D<8E `K?<a 8E; `8EK@;@J<JK89C@J?D<EK8Ie

@8E@JDa KF D<8E `8EK@;@J<JK89C@J?D<EK8I@8E@JDa @EJK<8; F= 8EK@;@Je<JK89C@J?D<EK8I@8E@JDFK?<IN8P8IFLE;

-%&6+-!,'/'*+

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 275/309

23. Mutual Information, and Density in Thingspace

Continuation of  EKIFGP8E;.?FIKF;<J

.LGGFJ< PFL ?8M< 8 JPJK<D 3 K?8K :8E 9< @E 8EP F= JK8K<J N?@:? 8I< 8CC <HL8CCP GIF989C< hI<C8K@M< KF PFLI :LII<EK JK8K< F= BEFNC<;><i 8E; 8 JPJK<D 4 K?8K :8E 9< @E 8EP F= JK8K<J 8CC <HL8CCP GIF989C<

/?< <EKIFGP F= 3 8J ;<=@E<; P<JK<I;8P @J 9@KJ N<cCC E<<; KF8JB P<JeFIeEF HL<JK@FEJ KF =@E; FLK 3cJ <O8:K JK8K< /?< <EKIFGP F= 4 8J ;<=@E<; P<JK<I;8P @J 9@KJ N< ?8M< KF 8JB P<JeFIeEF HL<Je

K@FEJ KF =@E; FLK 4cJ <O8:K JK8K< /?@J D8P J<<D F9M@FLJ J@E:<

8E; JF HL<JK@FEJ :8E ;@JK@E>L@J? GFJJ@9@C@K@<J 8E; HL<JK@FEJ :8E ;@JK@E>L@J? GFJJ@9@C@K@<J 9LK I<D<D9<I K?8K @= K?<GFJJ@9@C@K@<J N<I< EFK 8CC <HL8CCP C@B<CP N< :FLC; LJ< 8 DFI< :C<M<I:F;< KF ;@J:FM<I 4cJ JK8K< LJ@E> <> HL<JK@FEJ FE 8M<I8>< $EK?@J :8J< K?FL>? 3cJ probability mass @J evenly distributed FM<I 8CC @KJGFJJ@9C<JK8K<J8E;C@B<N@J<4JFN<:8EcKLJ<8EP:C<M<I:F;<J

2?8K@JK?<<EKIFGPF=K?<:FD9@E<;JPJK<Dh34i

4FL D@>?K 9< K<DGK<; KF 8EJN<I `$K K8B<J HL<JK@FEJ KF =@E;FLK 3 8E; K?<E HL<JK@FEJ KF =@E; FLK 4 JF @K K8B<J HL<JK@FEJKFK8CKF=@E;FLKK?<JK8K<F=38E;4a

LK N?8K @= K?< KNF M8I@89C<J 8I< <EK8E>C<; JF K?8K C<8IE@E> K?<JK8K<F=4K<CCJLJJFD<K?@E>89FLKK?<JK8K<F=3

$E G8IK@:LC8I C<KcJ JLGGFJ< K?8K 3 8E; 4 8I< <@K?<I 9FK? F;; FI9FK?<M<E

)FN @= N< I<:<@M< 8 e9@K D<JJ8>< h8JB HL<JK@FEJi 8E; C<8IE

K?8K 3 @J @E JK8K< N< BEFN K?8K 4 @J @E JK8K< FI JK8K< 9LK EFKJK8K< FI JK8K< .F K?< J@E>C< 8;;@K@FE8C HL<JK@FE `$J 4 @E JK8K<a 8EJN<I<; `)Fa K<CCJ LJ K?< <EK@I< JK8K< F= h34i 33 44

 E;N<C<8IE<;K?@JN@K?8KFK8CF=HL<JK@FEJ

FEM<IJ<CP @= N< C<8IE K?8K 4 @J @E JK8K< LJ@E> KNF HL<JK@FEJ@K N@CC K8B< LJ FECP 8E 8;;@K@FE8C KNF HL<JK@FEJ KF C<8IE N?<K?<I 3@J @E JK8K< FI >8@E =FLI HL<JK@FEJ KF C<8IE K?< JK8K< F= K?<AF@EKJPJK<D

http://lesswrong.com/lw/o2/mutual_information_and_density_in_thingspace/  +8><bEKIFGP8E;.?FIKF;<Jc

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 276/309

/?< mutual information F= KNF M8I@89C<J @J ;<=@E<; 8J K?< ;@==<Ie<E:< 9<KN<<E K?< <EKIFGP F= K?< AF@EK JPJK<D 8E; K?< <EKIFGP F= K?<@E;<G<E;<EKJPJK<DJ $h34i#h3i#h4ie#h34i

#<I< K?<I< @J FE< 9@K F= DLKL8C @E=FID8K@FE 9<KN<<E K?< KNFJPJK<DJ '<8IE@E> 3 K<CCJ LJ FE< 9@K F= @E=FID8K@FE 89FLK 4 h:LKJ;FNE K?< JG8:< F= GFJJ@9@C@K@<J =IFD KF 8 =8:KFIeF=e ;<:I<8J< @EK?< MFCLD<i 8E; C<8IE@E> 4 K<CCJ LJ FE< 9@K F= @E=FID8K@FE 89FLK 3h:LKJ;FNEK?<GFJJ@9@C@KPJG8:<=IFDKFi

2?8K 89FLK N?<E GIF989@C@KP D8JJ @J EFK <M<ECP ;@JKI@9LK<;4<JK<I;8P =FI <O8DGC< N< ;@J:LJJ<; K?< :8J< @E N?@:? 4 ?8; K?<GIF989@C@K@<J =FI @KJ =FLI JK8K<J '<K LJ K8B< K?@J KF9< FLI GIF989@C@KP ;@JKI@9LK@FE FM<I 4 :FEJ@;<I<; @E;<G<E;<EKCP e

@= N< J8N 4 N@K?FLK J<<@E> 8EPK?@E> <CJ< K?@J @J N?8K N<c; <OG<:KKF J<< E; JLGGFJ< K?< M8I@89C< 5 ?8J KNF JK8K<J 8E; N@K?GIF989@C@K@<J8E;I<JG<:K@M<CP

/?<E @= 8E; FECP @= K?< AF@EK ;@JKI@9LK@FE F= 4 8E; 5 @J 8J =FCCFNJK?<I<@JQ<IFDLKL8C@E=FID8K@FE9<KN<<E48E;5

54 54 54 54

54 54 54 54

/?@J;@JKI@9LK@FEF9<PJK?<C8N

Gh45i+h4i+h5i

!FI<O8DGC<+h54 i+h5 i+h4 i

 E; F9J<IM< K?8K N< :8E I<:FM<I K?< D8I>@E8C h@E;<G<E;<EKiGIF989@C@K@<JF=48E;5ALJK9PCFFB@E>8KK?<AF@EK;@JKI@9LK@FE

+h4 iKFK8CGIF989@C@KPF=8CCK?<;@==<I<EKN8PJ4 :8E?8GG<E+h54 i+h54 i

.F ALJK 9P @EJG<:K@E> K?< AF@EK ;@JKI@9LK@FE N< :8E ;<K<ID@E< N?<K?<I K?< D8I>@E8C M8I@89C<J 4 8E; 5 8I< @E;<G<E;<EK K?8K @J

-%&6+-!,'/'*+

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 277/309

 N?<K?<I K?< AF@EK ;@JKI@9LK@FE =8:KFIJ @EKF K?< GIF;L:K F= K?<D8I>@E8C ;@JKI@9LK@FEJ N?<K?<I =FI 8CC 4 8E; 5 +h45i +h4i+h5i

/?@JC8JK@JJ@>E@=@:8EK9<:8LJ<9P 8P<JcJ-LC<

+h4@5A i+h4@ i+h5A i+h4@5A i+h5A i+h4@ i+h4@R5A i+h4@ i

$E E>C@J? `=K<I PFL C<8IE 5A PFLI 9<C@<= 89FLK 4@ @J ALJK N?8K@KN8J9<=FI<a

.F N?<E K?< ;@JKI@9LK@FE =8:KFI@Q<J e N?<E +h45i +h4i+h5i eK?@J @J equivalent KF `'<8IE@E> 89FLK 4 E<M<I K<CCJ LJ 8EPK?@E> 89FLK5FIM@:<M<IJ8a

!IFD N?@:? PFL D@>?K JLJG<:K :FII<:KCP K?8K K?<I< @J EF DLeKL8C @E=FID8K@FE 9<KN<<E 4 8E; 5 2?<I< K?<I< @J EF DLKL8C@E=FID8K@FEK?<I<@JEF8P<J@8E<M@;<E:<8E;M@:<M<IJ8

.LGGFJ< K?8K @E K?< ;@JKI@9LK@FE 45 89FM< N< KI<8K<; <8:?GFJJ@9C< :FD9@E8K@FE F= 4 8E; 5 8J 8 J<G8I8K< <M<EKgJF K?8KK?< ;@JKI@9LK@FE 45 NFLC; ?8M< 8 KFK8C F= GFJJ@9@C@K@<J N@K? K?<GIF989@C@K@<J J?FNEg8E; K?<E N< :8C:LC8K<; K?< <EKIFGP F= K?< ;@Je

KI@9LK@FE 45 K?< J8D< N8P N< NFLC; :8C:LC8K< K?< <EKIFGP F= 8EP ;@JKI@9LK@FE

CF> hiCF> hiCF> hi_CF> hi

4FL NFLC; <E; LG N@K? K?< J8D< KFK8C PFL NFLC; ><K @= PFL J<Ge8I8K<CP :8C:LC8K<; K?< <EKIFGP F= 4 GCLJ K?< <EKIFGP F= 5 /?<I< @JEF DLKL8C @E=FID8K@FE 9<KN<<E K?< KNF M8I@89C<J JF FLI LE:<IK8@Ee

KP 89FLK K?< AF@EK JPJK<D @J EFK 8EP C<JJ K?8E FLI LE:<IK8@EKP 89FLKK?< KNF JPJK<DJ :FEJ@;<I<; J<G8I8K<CP h$ 8D EFK J?FN@E> K?< :8Ce:LC8K@FEJ 9LK PFL 8I< N<C:FD< KF ;F K?<D 8E; $ 8D EFK J?FN@E> K?< GIFF= K?8K K?@J @J KIL< @E ><E<I8C 9LK PFL 8I< N<C:FD< KF "FF>C<FE`.?8EEFE<EKIFGPa8E;`DLKL8C@E=FID8K@FEai

2?8K@=K?<AF@EK;@JKI@9LK@FE;F<JEcK=8:KFI@Q< !FI<O8DGC<

54 54 54 54

+8><bE$EKL@K@M<OGC8E8K@FEF=8P<J/?<FI<Dc

%-,-$! &'*%,!'&&&+!,1!&,!&+(

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 278/309

54 54 54 54

$= PFL 8;; LG K?< AF@EK GIF989@C@K@<J KF ><K D8I>@E8C GIF989@Ce

@K@<J PFL J?FLC; =@E; K?8K +h4 i +h5 i 8E; JF FE e K?<D8I>@E8CGIF989@C@K@<J8I<K?<J8D<8J9<=FI<

LK K?< AF@EK GIF989@C@K@<J ;F EFK 8CN8PJ <HL8C K?< GIF;L:K F= K?< D8I>@E8C GIF989@C@K@<J !FI <O8DGC< K?< GIF989@C@KP +h54 i<HL8CJ N?<I< +h5 i+h4 i NFLC; <HL8C /?8K @JK?< GIF989@C@KP F= ILEE@E> @EKF 54 KF><K?<I @J >I<8K<I K?8E PFLc;<OG<:K 98J<; FE K?< GIF989@C@K@<J F= ILEE@E> @EKF 5 FI 4 J<G8I8K<eCP

2?@:?@EKLIE@DGC@<J

+h54 i+h5 i+h4 i+h54 i+h4 i+h5 i+h5R4 i+h5 i

.@E:< K?<I<cJ 8E `LELJL8CCP ?@>?a GIF989@C@KP =FI +h54 i e ;<e=@E<; 8J 8 GIF989@C@KP ?@>?<I K?8E K?< D8I>@E8C GIF989@C@K@<J NFLC;@E;@:8K< 9P ;<=8LCK e @K =FCCFNJ K?8K F9J<IM@E> 4 @J <M@;<E:< N?@:?

@E:I<8J<J K?< GIF989@C@KP F= 5 E; 9P 8 JPDD<KI@:8C 8I>LD<EKF9J<IM@E>5 DLJK=8MFI4

 J K?<I< 8I< 8K C<8JK JFD< M8CL<J F= 4 K?8K K<CC LJ 89FLK 5h8E; M@:< M<IJ8i K?<I< DLJK 9< DLKL8C @E=FID8K@FE 9<KN<<E K?< KNF

 M8I@89C<J 8E; JF PFL N@CC =@E;g$ 8D :FE=@;<EK K?FL>? $ ?8M<EcK 8:eKL8CCP :?<:B<;gK?8K :8C:LC8K@E> K?< <EKIFGP F= 45 P@<C;J C<JJ KFK8CLE:<IK8@EKP K?8E K?< JLD F= K?< @E;<G<E;<EK <EKIFG@<J F= 4 8E; 5#h45i #h4i #h5i e $h45i N@K? 8CC HL8EK@K@<J E<:<JJ8I@CP EFEe

E<>8K@M<

h$;@>I<JJ?<I<KFI<D8IBK?8KK?<JPDD<KIPF=K?<<OGI<JJ@FE=FIK?<DLKL8C@E=FID8K@FEJ?FNJK?8K4must K<CCLJ8JDL:?89FLK5FE8M<I8><8J5K<CCJLJ89FLK4 $C<8M<@K8J8E<O<I:@J<KFK?<I<8;<IKFI<:FE:@C<K?@J

 N@K?8EPK?@E>K?<PN<I<K8L>?K@ECF>@::C8JJ89FLK?FN@=8CCI8M<EJ8I<9C8:B9<@E>8CCFN<;KFI<8JFE-8M<EhOieC8:BhOi;F<JEcKD<8EPFLcI<8CCFN<;KFI<8JFEC8:BhOie

-8M<EhOi #FN;@==<I<EKJ<<DK?<JPDD<KI@:8C

-%&6+-!,'/'*+

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 279/309

GIF989@C@KP=CFNJF=K?<8P<J@8E=IFDK?<J?8IGCLI:?<JF=CF>@:g<M<EK?FL>?K?<C8KK<I@JALJK8;<><E<I8K<:8J<F=K?<=FID<Ii

`LKa PFL 8JB `N?8K ?8J 8CC K?@J KF ;F N@K? K?< GIFG<I LJ< F=  NFI;Ja

$E DGKP '89<CJ 8E; K?<E -<GC8:< K?< .PD9FC N@K? K?< .L9eJK8E:< N< J8N K?< K<:?E@HL< F= I<GC8:@E> 8 NFI; N@K? @KJ ;<=@E@K@FEeK?<<O8DGC<9<@E>>@M<E

 CCjDFIK8CS=<8K?<IJ9@G<;8Ck8I<DFIK8C.F:I8K<J@J8jDFIK8CS=<8K?<IJ9@G<;8Ck

/?<I<=FI<.F:I8K<J@JDFIK8C

2?P K?<E NFLC; PFL <M<E N8EK KF ?8M< 8 NFI; =FI `?LD8Ea2?PEFKALJKJ8P`.F:I8K<J@J8DFIK8C=<8K?<IC<JJ9@G<;a

<:8LJ< @KcJ ?<CG=LC KF ?8M< J?FIK<I NFI;J =FI K?@E>J K?8K PFL<E:FLEK<I F=K<E $= PFLI :F;< =FI ;<J:I@9@E> J@E>C< GIFG<IK@<J @J8CI<8;P <==@:@<EK K?<E K?<I< N@CC EFK 9< 8E 8;M8EK8>< KF ?8M@E> 8 JG<:@8C NFI; =FI 8 :FEALE:K@FE e C@B< `?LD8Ea =FI `DFIK8C =<8K?<IC<JJ9@G<;a e LEC<JJ K?@E>J K?8K 8I< DFIK8C and  =<8K?<IC<JJ and  9@G<;8C8I< =FLE; more often K?8E K?< D8I>@E8C GIF989@C@K@<J NFLC; C<8; PFLKF<OG<:K

$E <==@:@<EK :F;<J NFI; C<E>K? :FII<JGFE;J KF GIF989@C@KPgJFK?< :F;< =FI 54 N@CC 9< ALJK 8J CFE> 8J K?< :F;< =FI 5 GCLJ K?<:F;< =FI 4 LEC<JJ +h54 i +h5 i+h4 i @E N?@:? :8J< K?< :F;< =FIK?<NFI;:8E9<J?FIK<IK?8EK?<:F;<J=FI@KJG8IKJ

 E; K?@J @E KLIE :FII<JGFE;J <O8:KCP KF K?< :8J< N?<I< N< :8E@E=<I JFD< F= K?< GIFG<IK@<J F= K?< K?@E> =IFD J<<@E> @KJ FK?<I

GIFG<IK@<J $K DLJK 9< DFI< C@B<CP K?8E K?< ;<=8LCK K?8K =<8K?<IC<JJ9@G<;8CK?@E>JN@CC8CJF9<DFIK8C

*= :FLIJ< K?< NFI; `?LD8Ea I<8CCP ;<J:I@9<J D8EP D8EP DFI<GIFG<IK@<J e N?<E PFL J<< 8 ?LD8EeJ?8G<; <EK@KP K?8K K8CBJ 8E;

 N<8IJ :CFK?<J PFL :8E @E=<I N?FC< ?FJKJ F= 9@F:?<D@:8C 8E;8E8KFD@:8C 8E; :F>E@K@M< =8:KJ 89FLK @K /F I<GC8:< K?< NFI; `?LeD8Ea N@K? 8 ;<J:I@GK@FE F= <M<IPK?@E> N< BEFN 89FLK ?LD8EJ

 NFLC; I<HL@I< LJ KF JG<E; 8E @EFI;@E8K< 8DFLEK F= K@D< K8CB@E>

+8><bDGKP'89<CJc +8><b-<GC8:<K?<.PD9FCN@K?K?<.L9JK8E:<c

%-,-$! &'*%,!'&&&+!,1!&,!&+(

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 280/309

LK K?@J @J KIL< only 9<:8LJ< 8 =<8K?<IC<JJ K8CB@E> 9@G<; @J =8I DFI<C@B<CP K?8E ;<=8LCK KF 9< GF@JFE89C< 9P ?<DCF:B FI ?8M< 9IF8; E8@CJFI9<FM<I:FE=@;<EK

#8M@E> 8 NFI; =FI 8 K?@E> I8K?<I K?8E ALJK C@JK@E> @KJ GIFG<IK@<J@J 8 DFI< :FDG8:K :F;< GI<:@J<CP @E K?FJ< :8J<J N?<I< N< :8E @E=<IJFD< F= K?FJ< GIFG<IK@<J =IFD K?< FK?<I GIFG<IK@<J h2@K? K?< <Oe:<GK@FE G<I?8GJ F= M<IP GI@D@K@M< NFI;J C@B< `I<;a K?8K N< NFLC;LJ< KF J<E; 8E <EK@I<CP LE:FDGI<JJ<; ;<J:I@GK@FE F= FLI J<EJFIP <OeG<I@<E:<J LK 9P K?< K@D< PFL <E:FLEK<I 8 9L> FI <M<E 8 IF:B

 PFLcI< ;<8C@E> N@K? EFEJ@DGC< GIFG<IKP :FCC<:K@FEJ =8I 89FM< K?<GI@D@K@M<C<M<Ci

.F ?8M@E> 8 NFI; `N@>>@Ea =FI >I<<Ee<P<; 9C8:Be?8@I<; G<FGC<@J DFI< LJ<=LC K?8E ALJK J8P@E> `>I<<Ee<P<; 9C8:Be?8@I<; G<IJFEaGI<:@J<CPN?<E

"I<<Ee<P<;G<FGC<8I<DFI<C@B<CPK?8E8M<I8><KF9<9C8:Be?8@I<;h8E;M@:<M<IJ8iD<8E@E>K?8KN<:8EGIF989@C@JK@:8CCP@E=<I>I<<E<P<J=IFD9C8:B?8@IFIM@:<

 M<IJ8 or  2@>>@EJJ?8I<FK?<IGIFG<IK@<JK?8K:8E9<@E=<II<;8K

 >I<8K<IeK?8Ee;<=8LCKGIF989@C@KP $EK?@J:8J<N<?8M<KF

J<G8I8K<CPF9J<IM<K?<>I<<E<P<J8E;9C8:B?8@I9LKK?<E8=K<IF9J<IM@E>9FK?K?<J<GIFG<IK@<J@E;<G<E;<EKCPN<:8EGIF989@C@JK@:8CCP@E=<IFK?<IGIFG<IK@<JhC@B<8K8JK<=FIB<K:?LGi

*E< D8P <M<E :FEJ@;<I K?< 8:K F= ;<=@E@E> 8 NFI; 8J 8 GIFD@J<KF K?@J <==<:K /<CC@E> JFD<FE< `$ ;<=@E< K?< NFI; bN@>>@Ec KF D<8E8 G<IJFE N@K? >I<<E <P<J 8E; 9C8:B ?8@Ia 9P "I@:<8E @DGC@:8K@FE8JJ<IKJ K?8K K?< NFI; `N@>>@Ea N@CC JFD<?FN ?<CG PFL D8B< @E=<Ie<E:<JJ?FIK<EPFLID<JJ8><J

$= >I<<Ee<P<J 8E; 9C8:B ?8@I ?8M< EF >I<8K<I K?8E ;<=8LCK GIF9e89@C@KP KF 9< =FLE; KF><K?<I EFI ;F<J 8EP FK?<I GIFG<IKP F::LI 8K

 >I<8K<I K?8E ;<=8LCK GIF989@C@KP 8CFE> N@K? K?<D K?<E K?< NFI;`N@>>@Ea @J 8 C@< /?< NFI; :C8@DJ K?8K :<IK8@E G<FGC< 8I< NFIK? ;@JeK@E>L@J?@E>8J8>IFLG9LKK?<PcI<EFK

$E K?@J :8J< K?< NFI; `N@>>@Ea ;F<J EFK ?<CG ;<J:I@9< I<8C@KP DFI< :FDG8:KCPg@K @J EFK ;<=@E<; 9P JFD<FE< J<E;@E> K?< J?FIK<JKD<JJ8><g@K ?8J EF IFC< @E K?< J@DGC<JK <OGC8E8K@FE HL@M8C<EKCP

+8><b.E<8B@E>@EFEEFK8K@FEJc

-%&6+-!,'/'*+

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 281/309

K?< NFI; `N@>>@Ea N@CC 9< F= EF ?<CG KF PFL @E ;F@E> 8EP 8P<J@8E@E=<I<E:< M<E @= PFL ;F EFK :8CC K?< NFI; 8 C@< @K @J JLI<CP 8E <IIFI

 E; K?< N8P KF :8IM< I<8C@KP 8K @KJ AF@EKJ @J KF ;I8N PFLI 9FLE;e

8I@<J 8IFLE; :FE:<EKI8K@FEJ F= LELJL8CCP ?@>? GIF989@C@KP ;<EJ@KP @E/?@E>JG8:<

+8><b/?<CLJK<I.KIL:KLI<F=/?@E>JG8:<c

%-,-$! &'*%,!'&&&+!,1!&,!&+(

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 282/309

24. Superexponential Conceptspace, andSimple Words

Followup to (LKL8C$E=FID8K@FE8E;<EJ@KP@E/?@E>JG8:<

/?@E>JG8:< PFL D@>?K K?@EB @J 8 I8K?<I ?L>< JG8:< (L:?C8I><I K?8E I<8C@KP =FI N?<I< I<8C@KP FECP :FEK8@EJ K?@E>J K?8K 8:KLe8CCP<O@JK/?@E>JG8:<:FEK8@EJ<M<IPK?@E>K?8K could <O@JK

 :KL8CCP K?< N8P $ `;<=@E<;a /?@E>JG8:< KF ?8M< ;@D<EJ@FEJ =FI<M<IP GFJJ@9C< 8KKI@9LK<g@E:CL;@E> :FII<C8K<; 8KKI@9LK<J C@B< ;<EJ@eKP 8E; MFCLD< 8E; D8JJg/?@E>JG8:< D8P 9< KFF GFFICP ;<=@E<; KF?8M< 8EPK?@E> PFL :FLC; :8CC 8 size. LK @KcJ @DGFIK8EK KF 9< 89C< KF

 M@JL8C@Q< /?@E>JG8:< anyway. .LI<CP EF FE< :8E really LE;<IJK8E; 8 =CF:B F= JG8IIFNJ @= 8CC K?<P J<< @J 8 :CFL; F= =C8GG@E> :8N@E> K?@E>JI8K?<IK?8E8:CLJK<IF=GF@EKJ@E/?@E>JG8:<

LK 8J M8JK 8J /?@E>JG8:< D8P 9< @K ;F<JEcK ?FC; 8 :8E;C< KF K?<J@Q<F=FE:<GKJG8:<

`FE:<GKa @E D8:?@E< C<8IE@E> D<8EJ 8 ILC< K?8K @E:CL;<J FI<O:CL;<J <O8DGC<J $= PFL J<< K?< ;8K8 e e e K?<E

 PFL D@>?K >L<JJ K?8K K?< :FE:<GK N8J `<M<E ELD9<IJa /?<I< @J 8 

I8K?<I C8I>< C@K<I8KLI< h8J FE< D@>?K <OG<:Ki FE ?FN KF C<8IE :FEe:<GKJ =IFD ;8K8_ >@M<E I8E;FD <O8DGC<J >@M<E :?FJ<E <O8DGC<J_ >@M<E GFJJ@9C< <IIFIJ @E :C8JJ@=@:8K@FE_ 8E; DFJK @DGFIK8EKCP >@M<E;@==<I<EKJG8:<JF=GFJJ@9C<ILC<J

.LGGFJ< =FI <O8DGC< K?8K N< N8EK KF C<8IE K?< :FE:<GK `>FF;;8PJFEN?@:?KFGC8PK<EE@Ja /?<GFJJ@9C<8KKI@9LK<JF=8PJ8I<

,?M O,IBBM@CI9M+6>BMQ

>F-:AD O/6FAC@9Q

#IA>9>HMO'CFA6@#><=Q/>B9 O,HFCB</:6?Q

2<cI< K?<E GI<J<EK<; N@K? K?< =FCCFN@E> ;8K8 N?<I< @E;@:8K<J8 GFJ@K@M< <O8DGC< F= K?< :FE:<GK 8E; e @E;@:8K<J 8 E<>8K@M< :C8JJ@e=@:8K@FE

http://lesswrong.com/lw/o3/superexponential_conceptspace_and_simple_words/ 

+8><b(LKL8C$E=FID8K@FE8E;<EJ@KP@E/?@E>JG8:<c +8><b/?<CLJK<I.KIL:KLI<F=/?@E>JG8:<c

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 283/309

,?M,IBBM >F-:AD/6FA

#IA>9>HM#><= />B9,HFCB<

,?M+6>BM >F-:ADC@9

#IA>9>HM#><= />B9,HFCB< ,?M,IBBM >F-:AD/6FA

#IA>9>HM#><= />B9/:6?

2?8KJ?FLC;8E8C>FI@K?D@E=<I=IFDK?@J

  D8:?@E< C<8IE<I D@>?K I<GI<J<EK one :FE:<GK K?8K =@KJ K?@J ;8eK88J=FCCFNJ

,?M >F-:AD/6FA #IA>9>HM

#><= />B9

$E K?@J =FID8K KF ;<K<ID@E< N?<K?<I K?@J :FE:<GK 8::<GKJ FII<A<:KJ 8E <O8DGC< N< :FDG8I< <C<D<EKe9Pe<C<D<EK 8::<GKJ 8EPeK?@E>9LK8JG<:@=@:M8CL<8::<GKJFECPK?8KJG<:@=@:M8CL<

.F K?< :FE:<GK 89FM< N@CC 8::<GK FECP 8PJ N@K? @I/<DeG28ID 8E; #LD@;@KP#@>? 9LK K?< .BP 8E; K?< 2@E; :8E K8B<FE 8EP M8CL< /?@J =@KJ 9FK? K?< E<>8K@M< 8E; K?< GFJ@K@M< :C8JJ@=@:8e

K@FEJ @E K?< ;8K8 JF =8IgK?FL>? @K @JEcK K?< only :FE:<GK K?8K ;F<JJF

2< :8E 8CJF J@DGC@=P K?< 89FM< :FE:<GK I<GI<J<EK8K@FE KF l28ID#@>?m

2@K?FLK>F@E>@EKF;<K8@CJK?<:C8JJ@:8C>FI@K?DNFLC;9<

[ (8@EK8@EK?<J<KF=K?<DFJK><E<I8C?PGFK?<J<JK?8K=@KK?<;8K8gK?FJ<K?8KGFJ@K@M<CP:C8JJ@=P8JD8EP<O8DGC<J8JGFJJ@9C<N?@C<JK@CC=@KK@E>K?<=8:KJ

[ (8@EK8@E8EFK?<IJ<KF=K?<DFJKJG<:@=@:?PGFK?<J<JK?8K

=@KK?<;8K8gK?FJ<K?8KE<>8K@M<CP:C8JJ@=P8JD8EP <O8DGC<J8JGFJJ@9C<N?@C<JK@CC=@KK@E>K?<=8:KJ

[ 8:?K@D<N<J<<8E<NE<>8K@M<<O8DGC<N<JKI<E>K?<E8CCK?<DFJK><E<I8C?PGFK?<J<J8JC@KKC<8JGFJJ@9C<JFK?8KK?<E<NJ<K@J8>8@E8J><E<I8C8JGFJJ@9C<N?@C<=@KK@E>K?<=8:KJ

[ 8:?K@D<N<J<<8E<NGFJ@K@M<<O8DGC<N<I<C8O8CCK?<DFJKJG<:@=@:?PGFK?<J<J8JC@KKC<8JGFJJ@9C<JFK?8KK?<

+-(*0('&&,!$'&(,+(&+!%($/'*+

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 284/309

E<NJ<K@J8>8@E8JJG<:@=@:8JGFJJ@9C<N?@C<=@KK@E>K?<=8:KJ

[ 2<:FEK@EL<LEK@CN<?8M<FECP8J@E>C<?PGFK?<J@JC<=K

/?@JN@CC9<K?<8EJN<I if K?<K8I><K:FE:<GKN8J@EFLI?PGFK?<J@JJG8:<8K8CC$E K?< :8J< 89FM< K?< J<K F= DFJK ><E<I8C ?PGFK?<J<J NFLC; 9<

l 28ID m 8E; l.LEEP m N?@C< K?< J<K F= DFJK JG<:@=@: ?PeGFK?<J<J@JK?<J@E>C<D<D9<I l.LEEP28ID#@>?m

 EP FK?<I :FE:<GK PFL :8E =@E; K?8K =@KJ K?< ;8K8 N@CC 9< JKI@:KCP DFI< JG<:@=@: K?8E FE< F= K?< DFJK ><E<I8C ?PGFK?<J<J 8E; JKI@:KCP DFI<><E<I8CK?8EK?<DFJKJG<:@=@:?PGFK?<J@J

h!FI DFI< FE K?@J $ I<:FDD<E; /FD (@K:?<CCcJ Machine Learn-

ing, =IFDN?@:?K?@J<O8DGC<N8J8;8GK<;i)FN PFL D8P EFK@:< K?8K K?< =FID8K 89FM< cannot  I<GI<J<EK 8CC

GFJJ@9C< :FE:<GKJ > `+C8P K<EE@J N?<E K?< JBP @J JLEEP or K?< 8@I@J N8IDa /?8K =@KJ K?< ;8K8 9LK @E K?< :FE:<GK I<GI<J<EK8K@FE ;<e=@E<;89FM<K?<I<cJEFHL8;ILGC<KF=M8CL<JK?8K;<J:I@9<JK?<ILC<

C<8ICP FLI D8:?@E< C<8IE<I @J EFK M<IP ><E<I8C 2?P EFK 8CCFN @K KF I<GI<J<EK all possible :FE:<GKJ JF K?8K @K :8E C<8IE N@K? K?<

 >I<8K<JKGFJJ@9C<=C<O@9@C@KP

8PJ 8I< :FDGFJ<; F= K?<J< =FLI M8I@89C<J FE< M8I@89C< N@K?  M8CL<J 8E; K?I<< M8I@89C<J N@K? M8CL<J .F K?<I< 8I< GFJJ@9C<8PJK?8KN<:FLC;<E:FLEK<I

/?< =FID8K >@M<E =FI I<GI<J<EK@E> FE:<GKJ 8CCFNJ LJ KF I<HL@I<8EP F= K?<J< M8CL<J =FI 8 M8I@89C< FI C<8M< K?< M8I@89C< FG<E .FK?<I< 8I< :FE:<GKJ @E K?8K I<GI<J<EK8K@FE !FI K?<DFJKe><E<I8CDFJKeJG<:@=@: 8C>FI@K?D KF NFIB N< E<<; KF JK8IK

 N@K? K?< DFJK JG<:@=@: ?PGFK?<J@J `EF <O8DGC< @J <M<I GFJ@K@M<CP :C8JJ@=@<;a $=N<8;;K?8K@KD8B<J8KFK8CF=:FE:<GKJ

$J @K JLJG@:@FLJ K?8K K?<I< 8I< DFI< GFJJ@9C< :FE:<GKJ K?8E GFJeJ@9C< 8PJ .LI<CP EFK =K<I 8CC 8 :FE:<GK :8E 9< M@<N<; 8J 8 collection F= 8PJ :FE:<GK :8E 9< M@<N<; 8J K?< J<K F= ;8PJ K?8K @K:C8JJ@=@<J GFJ@K@M<CP FI @JFDFIG?@:8CCP K?< J<K F= ;8PJ K?8K @K :C8JJ@e=@<JE<>8K@M<CP

.F K?< JG8:< F= all possible :FE:<GKJ K?8K :C8JJ@=P 8PJ @J K?< J<K

F=8CCGFJJ@9C<J<KJF=8PJN?FJ<J@Q<@J

/?@J :FDGC<K< JG8:< @E:CL;<J 8CC K?< :FE:<GKJ N< ?8M< ;@J:LJJ<;

JF =8I LK @K 8CJF @E:CL;<J :FE:<GKJ C@B< `+FJ@K@M<CP :C8JJ@=P FECP K?<

-%&6+-!,'/'*+

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 285/309

<O8DGC<J l.LEEP 28ID #@>? .KIFE>m 8E; l.LEEP 28ID #@>?2<8Bm 8E; I<A<:K <M<IPK?@E> <CJ<a FI `)<>8K@M<CP :C8JJ@=P FECP K?<<O8DGC< l-8@EP FC; #@>? .KIFE>m 8E; 8::<GK <M<IPK?@E> <CJ<a $K

@E:CL;<J :FE:<GKJ N@K? EF :FDG8:K I<GI<J<EK8K@FE ALJK 8 =C8K C@JK F=  N?8K@J8E;@JEcK8CCFN<;

/?8KcJ K?< GIF9C<D N@K? KIP@E> KF 9L@C; 8 `=LCCP ><E<I8Ca @Ee;L:K@M< C<8IE<I /?<P :8EcK C<8IE :FE:<GKJ LEK@C K?<PcM< J<<E <M<IP GFJJ@9C<<O8DGC<@EK?<@EJK8E:<JG8:<

$= N< 8;; FE DFI< 8KKI@9LK<J KF 8PJgC@B< K?< 28K<I K<DG<Ie8KLI< FI K?< !FI<:8JK =FI KFDFIIFNgK?<E K?< ELD9<I F= GFJJ@9C<;8PJ N@CC >IFN <OGFE<EK@8CCP @E K?< ELD9<I F= 8KKI@9LK<J LK K?@J@JEcK 8 GIF9C<D N@K? FLI I<JKI@:K<; :FE:<GK JG8:< 9<:8LJ< PFL :8E

E8IIFN ;FNE 8 C8I>< JG8:< LJ@E> 8 CF>8I@K?D@: ELD9<I F= <O8DGC<J'<KcJ J8P N< 8;; K?< 28K<I l28ID FC;m 8KKI@9LK< KF ;8PJ

 N?@:? N@CC D8B< =FI GFJJ@9C< 8PJ 8E; GFJJ@9C< :FE:<GKJ'<KcJ J8P K?8K <8:? 8P N< J<< @J LJL8CCP :C8JJ@=@<; GFJ@K@M< 9P 8IFLE; ?8C= F= K?< :LII<EKCPeGC8LJ@9C< :FE:<GKJ 8E; :C8JJ@=@<; E<>8eK@M< 9P K?< FK?<I ?8C= /?<E N?<E N< C<8IE K?< 8:KL8C :C8JJ@=@:8K@FEF= K?< <O8DGC< @K N@CC :LK K?< JG8:< F= :FDG8K@9C< :FE:<GKJ @E ?8C=

.F @K D@>?K FECP K8B< <O8DGC<J h i KF E8IIFN GFJJ@9C<

:FE:<GKJ;FNEKFFE<M<E @= 8PJ ?8; =FIKP 9@E8IP 8KKI@9LK<J @K J?FLC; JK@CC FECP K8B<

8 D8E8><89C< 8DFLEK F= ;8K8 KF E8IIFN ;FNE K?< GFJJ@9C< :FE:<GKJKF FE< <O8DGC<J @= <8:? <O8DGC< @J :C8JJ@=@<; GFJ@K@M< 9P ?8C= K?< I<D8@E@E> :FE:<GKJ Assuming, F= :FLIJ< K?8K K?< actual ILC< @JFE<N<:8EI<GI<J<EK8K8CC

$= PFL N8EK KF K?@EB F= 8CC K?< GFJJ@9@C@K@<J N<CC >FF; CL:B N@K?K?8K /?< JG8:< F= all possible :FE:<GKJ >IFNJ super <OGFE<EK@8CCP @EK?<ELD9<IF=8KKI@9LK<J

P K?< K@D< PFLcI< K8CB@E> 89FLK ;8K8 N@K? =FIKP 9@E8IP 8KKI@9eLK<J K?< ELD9<I F= GFJJ@9C< <O8DGC<J @J G8JK 8 KI@CC@FEg9LK K?<ELD9<I F= GFJJ@9C< concepts @J G8JK KNFeKFeK?<eKI@CC@FEK?eGFN<I /FE8IIFN ;FNE K?8K super <OGFE<EK@8C :FE:<GK JG8:< PFLc; ?8M< KF J<<FM<I 8 KI@CC@FE <O8DGC<J 9<=FI< PFL :FLC; J8P N?8K N8J $E 8E; N?8K

 N8J*LK 4FLc;?8M<KFJ<<<M<IPGFJJ@9C<<O8DGC<@E=8:K

/?8KcJ N@K? =FIKP 9@E8IP 8KKI@9LK<J D@E; PFL 9@KJ FI 9PK<J KF 9< :C8JJ@=@<; J@DGCP `4<Ja FI `)Fa 9@KJ @DGC@<J 6

+-(*0('&&,!$'&(,+(&+!%($/'*+

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 286/309

GFJJ@9C< <O8DGC<J 8E; 6h6i GFJJ@9C< :FE:<GKJ K?8K :C8JJ@=P K?FJ<<O8DGC<J8JGFJ@K@M<FIE<>8K@M<

.F ?<I< @E K?< I<8C NFIC; N?<I< F9A<:KJ K8B< DFI< K?8E 9PK<J

KF ;<J:I@9< and  8 KI@CC@FE <O8DGC<J 8I< EFK 8M8@C89C< and  K?<I< @JEF@J< @E K?< KI8@E@E> ;8K8 N< FECP <M<E think 89FLK highly regular :FE:<GKJ ?LD8E D@E;gFI K?< N?FC< F9J<IM89C< LE@M<IJ<g@JEFKE<8ICPC8I><<EFL>?KF:FEJ@;<I8CCK?<FK?<I?PGFK?<J<J

!IFD K?@J G<IJG<:K@M< C<8IE@E> ;F<JEcK ALJK rely on @E;L:K@M<9@8J @K @J nearly all @E;L:K@M< 9@8JgN?<E PFL :FDG8I< K?< ELD9<IF=:FE:<GKJILC<;FLK a priori, KFK?FJ<ILC<;FLK9PD<I<<M@;<E:<.

LK N?8K ?8J K?@J hPFL @EHL@I<i KF ;F N@K? K?< GIFG<I LJ< F=  NFI;J

$KcJ K?< N?FC< I<8JFE K?8K NFI;J ?8M< @EK<EJ@FEJ 8J N<CC 8J <OeK<EJ@FEJ

$E P<JK<I;8PcJGFJK$:FE:CL;<;

/?<N8PKF:8IM<I<8C@KP8K@KJAF@EKJ@JKF;I8N 9FLE;8I@<J8IFLE;:FE:<EKI8K@FEJF=LELJL8CCP?@>?GIF989@C@KP;<EJ@KP

$ ;<C@9<I8K<CP C<=K FLK 8 B<P HL8C@=@:8K@FE @E K?8K hJC@>?KCP <;@K<;iJK8K<D<EK 9<:8LJ< $ :FLC;EcK <OGC8@E @K LEK@C KF;8P 9<KK<I JK8K<eD<EKNFLC;9<

/?<N8PKF:8IM<I<8C@KP8K@KJAF@EKJ@JKF;I8N simple9FLE;8I@<J8IFLE;:FE:<EKI8K@FEJF=LELJL8CCP?@>?GIF989@C@KP;<EJ@KP@E/?@E>JG8:<

*K?<IN@J< PFL NFLC; ALJK ><IIPD8E;<I /?@E>JG8:< 4FL NFLC;

:I<8K< I<8CCP F;; EFE:FEK@>LFLJ 9FLE;8I@<J K?8K :FCC<:K<; K?< F9eJ<IM<; <O8DGC<J <O8DGC<J K?8K :FLC;EcK 9< ;<J:I@9<; @E 8EP J?FIK<ID<JJ8>< K?8E PFLI F9J<IM8K@FEJ K?<DJ<CM<J 8E; J8P `/?@J @J N?8K$cM<J<<E9<=FI<8E;N?8K$<OG<:KKFJ<<DFI<F=@EK?<=LKLI<a

$E K?< I<8C NFIC; EFK?@E> 89FM< K?< C<M<C F= DFC<:LC<J I<G<8KJ@KJ<C=  exactly. .F:I8K<J @J J?8G<; 8 CFK C@B< 8CC K?FJ< FK?<I ?LD8EJ

http://lesswrong.com/lw/hg/inductive_bias/  +8><bOK<EJ@FEJ8E;$EK<EJ@FEJc

+8><b(LKL8C$E=FID8K@FE8E;<EJ@KP@E/?@E>JG8:<c +8><bEKIFGP8E;.?FIKF;<Jc

-%&6+-!,'/'*+

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 287/309

 N?F N<I< MLCE<I89C< KF ?<DCF:B 9LK ?< @JEcK J?8G<; exactly C@B<K?<D .F PFLI >L<JJ K?8K .F:I8K<J @J 8 `?LD8Ea I<C@<J FE ;I8N@E>  simple 9FLE;8I@<J 8IFLE; K?< ?LD8E :CLJK<I @E /?@E>JG8:< -8K?<I

K?8E `/?@E>J J?8G<; <O8:KCP C@B< jeD<>89PK< J?8G< JG<:@=@:8K@FE k8E; N@K? jCFKJ F= FK?<I :?8I8:K<I@JK@:Jk or <O8:KCP C@B< jeD<>89PK<J?8G< JG<:@=@:8K@FE k 8E; jCFKJ F= FK?<I :?8I8:K<I@JK@:Jka _ 8I< ?LeD8Ea

$= PFL ;FEcK ;I8N  simple 9FLE;8I@<J 8IFLE; PFLI <OG<I@<E:<J PFL :8EcK ;F @E=<I<E:< N@K? K?<D .F PFL KIP KF ;<J:I@9< `8IKa

 N@K? @EK<EJ@FE8C ;<=@E@K@FEJ C@B< `K?8K N?@:? @J @EK<E;<; KF @EJG@I<8EP :FDGC<O <DFK@FE =FI K?< J8B< F= @EJG@I@E> @Ka I8K?<I K?8E ALJKGF@EK@E>8K8CFE>C@JKF=K?@E>JK?8K8I<FI8I<EcK8IK

$E =8:K K?< 89FM< JK8K<D<EK 89FLK `?FN KF :8IM< I<8C@KP 8K@KJ AF@EKJa @J 8 9@K :?@:B<Ee8E;e<>>@J? 4FL :8EcK 8JJ<JJ K?< densityF= 8:KL8C F9J<IM8K@FEJ LEK@C PFLcM< 8CI<8;P ;FE< 8K C<8JK 8 C@KKC<:8IM@E> E; K?< GIF989@C@KP ;@JKI@9LK@FE :FD<J =IFD ;I8N@E> K?<9FLE;8I@<J EFK K?< FK?<I N8P 8IFLE;g@= PFL 8CI<8;P had K?< GIF9e89@C@KP ;@JKI@9LK@FE PFLc; ?8M< <M<IPK?@E> E<:<JJ8IP =FI @E=<I<E:<JFN?PNFLC;PFL9FK?<I;I8N@E>9FLE;8I@<J

 E; K?@J JL>><JKJ 8EFK?<IgP<J P<K 8EFK?<IgI<8JFE KF 9< JLJe

G@:@FLJ F= K?< :C8@D K?8K `PFL :8E ;<=@E< 8 NFI; 8EP N8P PFL C@B<a2?<E PFL :FEJ@;<I K?< JLG<I<OGFE<EK@8C J@Q< F= FE:<GKJG8:< @K9<:FD<J :C<8I K?8K J@E>C@E> FLK FE< G8IK@:LC8I :FE:<GK =FI :FEJ@;e<I8K@FE @J 8E 8:K F= EF JD8CC 8L;8:@KPgEFK ALJK =FI LJ 9LK =FI 8EP D@E;F=9FLE;<;:FDGLK@E>GFN<I

+I<J<EK@E> LJ N@K? K?< NFI; `N@>>@Ea ;<=@E<; 8J `8 9C8:Be?8@I<; >I<<Ee<P<; G<IJFEa N@K?FLK JFD< I<8JFE =FI I8@J@E>  this particular concept KF K?< C<M<C F= FLI ;<C@9<I8K< 8KK<EK@FE @J I8K?<I C@B< 8 ;<K<:eK@M< J8P@E> `2<CC $ ?8M<EcK K?< JC@>?K<JK J?I<; F= JLGGFIK FE< N8P 

FI K?< FK?<I =FI N?F :FLC;cM< DLI;<I<; K?FJ< FIG?8EJ_ EFK <M<E8E @EKL@K@FE D@E; PFL_ 9LK ?8M< N< :FEJ@;<I<; %F?E , 2@==C<e?<@DF=)FIBC<-;8J8JLJG<:Ka

+8><b2?<I<KFI8NK?<FLE;8IPc http://lesswrong.com/lw/jo/einsteins_arrogance/ 

+-(*0('&&,!$'&(,+(&+!%($/'*+

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 288/309

25. Conditional Independence, and NaiveBayes

Followup to .<8I:?@E>=FI8P<Je.KIL:KLI<

+I<M@FLJCP $ JGFB< F= DLKL8C @E=FID8K@FE 9<KN<<E 3 8E; 4$h34i N?@:? @J K?< ;@==<I<E:< 9<KN<<E K?< <EKIFGP  F= K?< AF@EKGIF989@C@KP ;@JKI@9LK@FE #h34i 8E; K?< <EKIFG@<J F= K?< D8I>@E8C;@JKI@9LK@FEJ#h3i#h4i

$ >8M< K?< <O8DGC< F= 8 M8I@89C< 3 ?8M@E> <@>?K JK8K<J  N?@:? 8I< 8CC <HL8CCP GIF989C< @= N< ?8M< EFK P<K <E:FLEK<I<; 8EP <M@;<E:< 8E; 8 M8I@89C< 4 N@K? JK8K<J N?@:? 8I< 8CC <HL8CCP 

GIF989C< @= N< ?8M< EFK P<K <E:FLEK<I<; 8EP <M@;<E:< /?<E @= N<:8C:LC8K< K?< D8I>@E8C <EKIFG@<J #h3i 8E; #h4i N< N@CC =@E; K?8K 3?8J9@KJF=<EKIFGP8E;4?8J9@KJ

#FN<M<I N< 8CJF BEFN K?8K 3 8E; 4 8I< 9FK? <M<E FI 9FK? F;;8E; K?@J @J 8CC N< BEFN 89FLK K?< I<C8K@FE 9<KN<<E K?<D .F =FI K?<AF@EK ;@JKI@9LK@FE h34i K?<I< 8I< FECP GFJJ@9C< JK8K<J 8CC <HL8CCP GIF989C< =FI 8 AF@EK <EKIFGP F= 9@KJ /?@J @J 8 e9@K <EKIFGP ;<=<:K:FDG8I<; KF 9@KJ F= <EKIFGP @= 3 8E; 4 N<I< @E;<G<E;<EK /?@J

<EKIFGP ;<=<:K @J K?< DLKL8C @E=FID8K@FE e K?< @E=FID8K@FE K?8K 3K<CCJ LJ 89FLK 4 FI M@:< M<IJ8 JF K?8K N< 8I< EFK 8J LE:<IK8@E 89FLKFE<8=K<I?8M@E>C<8IE<;K?<FK?<I

.LGGFJ< ?FN<M<I K?8K K?<I< <O@JKJ 8 K?@I; M8I@89C< 5 5 ?8JKNF JK8K<J `<M<Ea 8E; `F;;a G<I=<:KCP :FII<C8K<; KF K?< <M<EE<JJ FIF;;E<JJ F= h34i $E =8:K N<cCC JLGGFJ< K?8K 5 @J ALJK K?< HL<JK@FE`I<38E;4<M<EFIF;;a

$= N< ?8M< EF <M@;<E:< 89FLK 3 8E; 4 K?<E 5 @KJ<C= E<:<JJ8I@CP ?8J 9@K F= <EKIFGP FE K?< @E=FID8K@FE >@M<E /?<I< @J 9@K F= DLe

KL8C @E=FID8K@FE 9<KN<<E 5 8E; 3 8E; 9@K F= DLKL8C @E=FID8K@FE9<KN<<E 5 8E; 4 E; 8J GI<M@FLJCP EFK<; 9@K F= DLKL8C @Ee=FID8K@FE 9<KN<<E 3 8E; 4 .F ?FN DL:? <EKIFGP =FI K?< N?FC<JPJK<Dh345i 4FLD@>?KE8@M<CP<OG<:KK?8K

http://lesswrong.com/lw/o8/conditional_independence_and_naive_bayes/  http://lesswrong.com/lw/o7/searching_for_bayesstructure/ 

+8><b(LKL8C$E=FID8K@FE8E;<EJ@KP@E/?@E>JG8:<c +8><bEKIFGP8E;.?FIKF;<Jc

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 289/309

#h345i#h3i#h4i#h5ie$h35ie$h54ie$h34i

9LKK?@JKLIEJFLKEFKKF9<K?<:8J</?< AF@EK JPJK<D h345i FECP ?8J GFJJ@9C< JK8K<J e J@E:< 5 @JALJKK?<HL<JK@FE`I<34<M<EFIF;;aeJF#h345i9@KJ

LK@=PFL:8C:LC8K<K?<=FIDLC8ALJK>@M<EPFL><K

heeei9@KJ9@KJ 2-*)"

2?P <:8LJ< @= PFL ?8M< K?< DLKL8C @E=FID8K@FE 9<KN<<E 38E; 5 8E; K?< DLKL8C @E=FID8K@FE 9<KN<<E 5 8E; 4 K?8K D8P 

@E:CL;< JFD< F= K?< same DLKL8C @E=FID8K@FE K?8K N<cCC :8C:LC8K< <Oe@JKJ 9<KN<<E 3 8E; 4 $E K?@J :8J< =FI <O8DGC< BEFN@E> K?8K 3@J <M<E K<CCJ LJ K?8K 5 @J <M<E 8E; BEFN@E> K?8K 5 @J <M<E K<CCJ LJK?8K 4 @J <M<E 9LK K?@J @J K?< J8D< @E=FID8K@FE K?8K 3 NFLC; K<CCLJ 89FLK 4 2< ;FL9C<e:FLEK<; JFD< F= FLI BEFNC<;>< 8E; JF:8D<LGN@K?KFFC@KKC<<EKIFGP

/?<:FII<:K=FIDLC8@Jh$9<C@<M<i

#h345i#h3i#h4i#h5ie$h35ie$h54ie$h34R5i

#<I< K?< C8JK K<ID $h34 R 5i D<8EJ `K?< @E=FID8K@FE K?8K 3K<CCJ LJ 89FLK 4 >@M<E K?8K N< 8CI<8;P BEFN 5a $E K?@J :8J< 3;F<JEcK K<CC LJ 8EPK?@E> 89FLK 4 >@M<E K?8K N< 8CI<8;P BEFN 5 JFK?< K<ID :FD<J FLK 8J Q<IF e 8E; K?< <HL8K@FE >@M<J K?< :FII<:K 8EeJN<I /?<I<@JEcKK?8KE@:<

`)Fa PFL :FII<:KCP  I<GCP `=FI PFL ?8M< EFK KFC; D< ?FN KF cal-

culate $h34R5i FECP >@M<E D< 8 M<I98C 8I>LD<EK K?8K @K FL>?K KF 9<Q<IFa

2< :8C:LC8K< $h34R5i ALJK K?< N8P PFL NFLC; <OG<:K $h34i #h3i#h4ie#h34iJF

$h34R5i#h3R5i#h4R5ie#h34R5i

http://www.youtube.com/watch?v=tRVUOGUmxJI 

+8><b!8B<8LJ8C@KPc http://yudkowsky.net/bayes/technical.html 

'&!,!'&$! &(&&&&!.1+

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 290/309

 E; EFN $ JLGGFJ< PFL N8EK KF BEFN ?FN KF :8C:LC8K< K?<:FE;@K@FE8C<EKIFGP 2<CCK?< original =FIDLC8=FIK?<<EKIFGP@J

#h.i.LD@Gh.@ ieCF> hGh.@ ii

$= N< K?<E C<8IE<; 8 E<N =8:K 5 FLI I<D8@E@E> LE:<IK8@EKP 89FLK.NFLC;9<

#h.R5 i.LD@Gh.@R5 ieCF> hGh.@R5 ii

.F @= N<cI< >F@E> KF C<8IE 8 E<N =8:K 5 9LK N< ;FEcK BEFN N?@:?5 P<K K?<E FE 8M<I8>< N< <OG<:K KF 9< 8IFLE; K?@J LE:<IK8@E F= .

8=K<IN8I;

#h.R5i.LDAhGh5A i.LD@Gh.@R5A ieCF> hGh.@R5A iii

 E; K?8KcJ ?FN FE< :8C:LC8K<J :FE;@K@FE8C <EKIFG@<J =IFD N?@:?@EKLIEN<:8E><KK?<:FE;@K@FE8CDLKL8C@E=FID8K@FE

/?<I<8I< all sorts F=8E:@CC8IPK?<FI<DJ?<I<C@B<

#h3R4i#h34ie#h4i

8E;

@= $h35i 8E; $h43R5i K?<E $h34i

9LK$cDEFK>F@E>KF>F@EKFK?FJ<

`LKa PFL 8JB `N?8K ;F<J this ?8M< KF ;F N@K? K?< E8KLI< F=  NFI;J8E;K?<@I?@;;<E8P<J@8EJKIL:KLI<a

$ 8D ALJK JF unspeakably >C8; K?8K PFL 8JB<; K?8K HL<JK@FE 9<e:8LJ< $ N8J GC8EE@E> KF K<CC PFL N?<K?<I PFL C@B<; @K FI EFK LK=@IJKK?<I<8I<8:FLGC<DFI<GI<C@D@E8I@<J

4FL N@CC I<D<D9<IgP<J PFL will  I<D<D9<IgK?8K K?<I< @J 8 ;L8C@KP 9<KN<<E DLKL8C @E=FID8K@FE 8E; 8P<J@8E <M@;<E:< (LKLe8C @E=FID8K@FE @J GFJ@K@M< @= 8E; FECP @= K?< GIF989@C@KP F= 8K C<8JKJFD< AF@EK <M<EKJ +hO Pi ;F<J EFK <HL8C K?< GIF;L:K F= K?< GIF9e89@C@K@<J F= K?< J<G8I8K< <M<EKJ +hOi+hPi /?@J @E KLIE @J <O8:KCP <HL@M8C<EK KF K?< :FE;@K@FE K?8K 8P<J@8E <M@;<E:< <O@JKJ 9<KN<<E O 

8E;P

-%&6+-!,'/'*+

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 291/309

$h34i +hOPi+hOi+hPi+hOPi+hPi+hOi

+hORPi+hOi

$= PFLcI< :FE;@K@FE@E> FE 5 PFL ALJK 8;ALJK K?< N?FC< ;<I@M8K@FE8::FI;@E>CP

$h34R5i +hOPRQi+hORQi+hPRQi+hOPRQi+hPRQi+hORQih+hOPQi+hQiih+hPQi+hQii+hORQi

+hOPQi+hPQi+hORQi+hORPQi+hORQi

2?@:? C8JK C@E< I<8;J `M<E BEFN@E> 5 C<8IE@E> 4 JK@CC :?8E><JFLI9<C@<=J89FLK3a

FEM<IJ<CP 8J @E FLI FI@>@E8C :8J< F= 5 9<@E> `<M<Ea FI `F;;a 5J:I<<EJ F==  3 =IFD 4 e K?8K @J @= N< BEFN K?8K 5 @J `<M<Ea C<8IE@E> K?8K 4 @J @E JK8K< K<CCJ LJ nothing more 89FLK N?<K?<I 3 @J FI *I @= N< BEFN K?8K 5 @J `F;;a K?<E C<8IE@E> K?8K 3 @J K<CCJ LJ

EFK?@E> DFI< 89FLK N?<K?<I 4 @J FI '<8IE@E> 5 ?8J I<E;<I<;38E;4 conditionally independent 

FE;@K@FE8C @E;<G<E;<E:< @J 8 ?L><CP @DGFIK8EK :FE:<GK @EGIF989@C@KP K?<FIPgKF :@K< ALJK FE< <O8DGC< N@K?FLK :FE;@K@FE8C@E;<G<E;<E:<K?<LE@M<IJ<NFLC;?8M<EFJKIL:KLI<

/F;8P K?FL>? $ FECP @EK<E; KF K8CB 89FLK FE< G8IK@:LC8I B@E;F= :FE;@K@FE8C @E;<G<E;<E:<gK?< :8J< F= 8 :<EKI8C M8I@89C< K?8KJ:I<<EJ F== FK?<I M8I@89C<J JLIIFLE;@E> @K C@B< 8 :<EKI8C 9F;P N@K?

K<EK8:C<J'<K K?<I< 9< =@M< M8I@89C<J 0 1 2 3 4 8E; DFI<FM<I JLGe

GFJ< K?8K =FI <M<IP G8@I F= K?<J< M8I@89C<J FE< M8I@89C< @J <M@;<E:<89FLK K?< FK?<I $= PFL J<C<:K 0 8E; 2 =FI <O8DGC< K?<E C<8IE@E> 00 N@CC K<CC PFL JFD<K?@E> PFL ;@;EcK BEFN 9<=FI< 89FLK K?<GIF989@C@KP22

 E LED8E8><89C< @E=<I<EK@8C D<JJ M@;<E:< >FE< N@C; )FKE<:<JJ8I@CP

+8><bI>LD<EK.:I<<EJ*==LK?FI@KPc

'&!,!'&$! &(&&&&!.1+

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 292/309

(8P9< 0 @J `.G<8BJ 8 C8E>L8><a 1 @J `/NF 8IDJ 8E; K<E ;@>@KJa2 @J `2<8IJ :CFK?<Ja 3 @J `+F@JFE89C< 9P ?<DCF:Ba 8E; 4 @J `-<;9CFF;a )FN @= PFL <E:FLEK<I 8 K?@E>e@EeK?<eNFIC; K?8K D@>?K 9<

8E 8GGC< 8E; D@>?K 9< 8 IF:B 8E; PFL C<8IE K?8K K?@J K?@E> JG<8BJ?@E<J< PFL 8I< C@89C< KF 8JJ<JJ 8 DL:? ?@>?<I GIF989@C@KP K?8K @K N<8IJ :CFK?<J 8E; @= PFL C<8IE K?8K K?< K?@E> @J EFK GF@JFE89C< 9P ?<DCF:B PFL N@CC 8JJ<JJ 8 JFD<N?8K CFN<I GIF989@C@KP K?8K @K ?8JI<;9CFF;

)FN JFD< F= K?<J< ILC<J 8I< JKIFE><I K?8E FK?<IJ /?<I< @J K?<:8J< F= !I<; N?F @J D@JJ@E> 8 =@E><I ;L< KF 8 MFC:8EF 8::@;<EK 8E;K?< :8J< F= 8IE<P K?< 89P N?F ;F<JEcK JG<8B P<K 8E; K?< :8J<F= $IM@E> K?< $-FK N?F <D@KJ J<EK<E:<J 9LK ?8J EF 9CFF; .F

@= N< C<8IE K?8K 8 :<IK8@E K?@E> @J EFK N<8I@E> :CFK?<J K?8K ;F<JEcKJ:I<<E F== <M<IPK?@E> K?8K @KJ JG<<:? :8G89@C@KP :8E K<CC LJ 89FLK @KJ9CFF; :FCFI $= K?< K?@E> ;F<JEcK N<8I :CFK?<J 9LK does K8CB D8P9<@KcJ)L;<)<CC@<

/?@J D8B<J K?< :8J< DFI< @EK<I<JK@E> K?8E J8P =@M< @EK<><I M8I@e89C<J K?8K 8I< 8CC F;; FI 8CC <M<E 9LK FK?<IN@J< LE:FII<C8K<; $EK?8K :8J< BEFN@E>  any FE< F= K?< M8I@89C<J NFLC; J:I<<E F== <Me<IPK?@E> K?8K BEFN@E> 8 J<:FE; M8I@89C< :FLC; K<CC LJ 89FLK 8 K?@I;

 M8I@89C<

LK ?<I< N< ?8M< ;<G<E;<E:@<J K?8K ;FEcK >F 8N8P 8J JFFE 8J N< C<8IE ALJK FE< M8I@89C< 8J K?< :8J< F= )L;< )<CC@< J?FNJ .F @J@K8ELED8E8><89C<@E=<I<EK@8C@E:FEM<E@<E:<

!<8I EFK =FI K?<I< D8P 9< JFD< sixth M8I@89C< 5 N?@:? @= N<BE<N @K I<8CCP  would  J:I<<E F== <M<IP G8@I F= M8I@89C<J =IFD <8:?FK?<I /?<I< D8P 9< JFD< M8I@89C< 5g<M<E @= N< ?8M< KF construct 5I8K?<IK?8EF9J<IM@E>@K;@I<:KCPgJL:?K?8K

GhLRMNOPQiGhLRQiGhMRLNOPQiGhMRQiGhNRLMOPQiGhNRQi

_

+<I?8GJ given that 8 K?@E> @J `?LD8Ea K?<E K?< GIF989@C@K@<J F= @K JG<8B@E> N<8I@E> :CFK?<J 8E; ?8M@E> K?< JK8E;8I; ELD9<I F= =@Ee

 ><IJ 8I< 8CC @E;<G<E;<EK !I<; D8P 9< D@JJ@E> 8 =@E><I e 9LK ?< @J EFDFI< C@B<CP KF 9< 8 EL;@JK K?8E K?< E<OK G<IJFE )L;< )<CC@< E<M<I

 N<8IJ :CFK?<J 9LK BEFN@E> K?@J ;F<JEcK D8B< @K 8EP C<JJ C@B<CP K?8K

-%&6+-!,'/'*+

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 293/309

J?< JG<8BJ 8E; 89P 8IE<P ;F<JEcK K8CB P<K 9LK @J EFK D@JJ@E> 8EP C@D9J

/?@J @J :8CC<; K?< `)8@M< 8P<Ja D<K?F; 9<:8LJ< @K LJL8CCP @JEcK

HL@K< KIL< 9LK pretending K?8K @KcJ KIL< :8E J@DGC@=P K?< C@M@E> ;8PeC@>?KJ FLK F= PFLI :8C:LC8K@FEJ 2< ;FEcK B<<G J<G8I8K< KI8:B F= K?<@E=CL<E:< F= :CFK?<;eE<JJ FE JG<<:? :8G89@C@KP >@M<E =@E><I ELD9<I2< ALJK LJ< 8CC K?< @E=FID8K@FE N<cM< F9J<IM<; KF B<<G KI8:B F= K?<GIF989@C@KP K?8K K?@J K?@E>P @J 8 ?LD8E hFI 8CK<IE8K@M<CP JFD<K?@E> <CJ< C@B< 8 :?@DG8EQ<< FI IF9FKi 8E; K?<E LJ< FLI 9<C@<=J 89FLK K?<:<EKI8C :C8JJ KF GI<;@:K 8EPK?@E> N< ?8M<EcK J<<E P<K C@B< MLCE<I8e9@C@KPKF?<DCF:B

 EP F9J<IM8K@FEJ F= 0 1 2 3 8E; 4 ALJK 8:K 8J <M@;<E:<

=FI K?< :<EKI8C :C8JJ M8I@89C< 5 8E; K?<E N< LJ< K?< GFJK<I@FI ;@JeKI@9LK@FE FE 5 KF D8B< 8EP GI<;@:K@FEJ K?8K E<<; D8B@E> 89FLKLEF9J<IM<;M8I@89C<J@E01238E;4

.FLE;=8D@C@8I $KJ?FLC;

C<>>

 J 8 D8KK<I F= =8:K @= PFL LJ< K?< I@>?K B@E; F= E<LI8C E<KNFIBLE@KJ K?@J `E<LI8C E<KNFIBa <E;J LG exactly, mathematically <HL@M8eC<EK KF )8@M< 8P<J /?< :<EKI8C LE@K ALJK E<<;J 8 CF>@JK@: K?I<J?eFC;g8E .e:LIM< I<JGFEJ<g8E; K?< N<@>?KJ F= K?< @EGLKJ ALJK E<<;KF D8K:? K?< CF>8I@K?DJ F= K?< C@B<C@?FF; I8K@FJ <K:<K<I8 $E =8:K

@KcJ 8 >FF; >L<JJ K?8K K?@J @J FE< F= K?< I<8JFEJ N?P CF>@JK@: I<JGFEJ<

'&!,!'&$! &(&&&&!.1+

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 294/309

F=K<E NFIBJ JF N<CC @E E<LI8C E<KNFIBJg@K C<KJ K?< 8C>FI@K?D JE<8B@E8C@KKC<8P<J@8EI<8JFE@E>N?@C<K?<;<J@>E<IJ8I<EcKCFFB@E>

 %LJK 9<:8LJ< JFD<FE< @J GI<J<EK@E> PFL N@K? 8E 8C>FI@K?D K?8K

K?<P :8CC 8 `E<LI8C E<KNFIBa N@K? 9LQQNFI;J C@B< `J:IL==Pa 8E;`<D<I><EKa GC8JK<I<; 8CC FM<I @K ;@J:C8@D@E> GIFL;CP K?8K K?<P ?8M<EF @;<8 ?FN K?< C<8IE<; E<KNFIB NFIBJgN<CC ;FEcK 8JJLD< K?8KK?<@I C@KKC< $ 8C>FI@K?D really is <PFE; K?< -<8CDJ F= 'F>@:!FI K?@J G8I8;@>D F= 8;?F:B<IP @= @K NFIBJ N@CC KLIE FLK KF ?8M<8P<J@8E JKIL:KLI< @K D8P <M<E 9< <O8:KCP <HL@M8C<EK KF 8E 8C>FeI@K?DF=K?<JFIK:8CC<;`8P<J@8Ea

M<E@=@K;F<JEcK look 8P<J@8EFEK?<JLI=8:<

 E; K?<E PFL ALJK know K?8K K?< 8P<J@8EJ 8I< >F@E> KF JK8IK<OGC8@E@E> <O8:KCP ?FN K?< 8C>FI@K?D NFIBJ N?8K LE;<ICP@E> 8JeJLDGK@FEJ @K I<=C<:KJ N?@:? <EM@IFED<EK8C I<>LC8I@K@<J @K <OGCF@KJ

 N?<I< @K NFIBJ 8E; N?<I< @K =8@CJ 8E; <M<E 8KK8:?@E> LE;<IJK8E;e89C<D<8E@E>JKFK?<C<8IE<;E<KNFIBN<@>?KJ

@J8GGF@EK@E>@JEcK@K

http://lesswrong.com/lw/o7/searching_for_bayesstructure/  +8><b(LKL8C$E=FID8K@FE8E;<EJ@KP@E/?@E>JG8:<c

-%&6+-!,'/'*+

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 295/309

26. Words as Mental Paintbrush Handles

Followup to FE;@K@FE8C$E;<G<E;<E:<8E;)8@M<8P<J

h2< J?FLC; 9< ;FE< N@K? K?< D8K?P GFJKJ $ K?@EB 8K C<8JK =FIEFN LK =FI>@M< D< @= @IFE@:8CCP $ <E; LG I<JFIK@E> KF -8K@FE8C@KP ,LFK<J =FI 8 ;8P FI KNF $cD :LII<EKCP 8K K?< "$e :FE=<I<E:<

 N?@:?8JF=K?<=@IJKJ<JJ@FE@JEFKE<8ICPJF98;8J$=<8I<;i

.LGGFJ< $ K<CC PFL `$KcJ K?< JKI8E><JK K?@E> /?< C8DGJ @E K?@J?FK<C?8M<KI@8E>LC8IC@>?K9LC9Ja

4FL D8P FI D8P EFK ?8M< M@JL8C@Q<; @Kg@= PFL ?8M<EcK ;FE< @K P<K ;F JF EFNgN?8K @E PFLI D@E;cJ <P< ;F<J 8 `KI@8E>LC8I C@>?Ke

9LC9aCFFBC@B<$EPFLID@E;cJ<P<;@;K?<>C8JJ?8M<J?8IG<;><JFIJDFFK?

2?<E K?< G?I8J< `KI@8E>LC8I C@>?K9LC9a =@IJK :IFJJ<; DP D@E;gEF K?< ?FK<C ;F<JEcK ?8M< K?<DgK?<E 8J 9<JK 8J DP @EKIFeJG<:K@FE :FLC; ;<K<ID@E< $ =@IJK J8N 8 GPI8D@;8C C@>?K9LC9 N@K?J?8IG <;><J K?<E h8CDFJK @DD<;@8K<CPi K?< <;><J N<I< JDFFK?<;8E; K?<E DP D@E; ><E<I8K<; 8 CFFG F= =CFLI<J:<EK 9LC9 @E K?< J?8G<F=8JDFFK?KI@8E>C<8J8E8CK<IE8K@M<

 J =8I 8J $ :8E K<CC EF ;<C@9<I8K@M<M<I98C K?FL>?KJ N<I< @Ee MFCM<;gALJK NFI;C<JJ I<=C<O =C@E:? 8N8P =IFD K?< @D8>@E8IP D<EK8C M@J@FE F= J?8IG >C8JJ N?@:? ;<J@>E GIF9C<D N8J JFCM<; 9<=FI< $:FLC;<M<EK?@EB@ENFI;J

<C@<M< @K FI EFK =FI JFD< ;<:8;<J K?<I< N8J 8 J<I@FLJ ;<98K<89FLK N?<K?<I G<FGC< really ?8; D<EK8C @D8><J @E K?<@I D@E;g8E8:KL8C picture F= 8 :?8@I JFD<N?<I<gFI @= G<FGC< ALJK E8@M<CP thought K?<P ?8; D<EK8C @D8><J h?8M@E> 9<<E D@JC<; 9P `@EKIFJG<:K@FEa 8 

 M<IP 98; =FI9@;;<E 8:K@M@KPi N?@C< 8:KL8CCP ALJK ?8M@E> 8 C@KKC< `:?8@Ia

C89<CC@B<8'$.+KFB<E8:K@M<@EK?<@I9I8@E$ 8D KIP@E> ?8I; EFK KF J8P 8EPK?@E> C@B< `#FN JG<:K8:LC8ICP J@Ce

CPa 9<:8LJ< K?<I< @J 8CN8PJ K?< ?@E;J@>?K <==<:K KF :FEJ@;<I 9LK?FNJG<:K8:LC8ICPJ@CCP

/?@J 8:8;<D@: G8I8;@>D $ K?@EB N8J DFJKCP 8 ;<I8E><; C<>8:P F= 9<?8M@FI@JD N?@:? ;<E@<; K?< <O@JK<E:< F= K?FL>?KJ @E ?LD8EJ8E; JFL>?K KF <OGC8@E 8CC ?LD8E G?<EFD<E8 8J `I<=C<Oa @E:CL;@E> 

http://lesswrong.com/lw/o9/words_as_mental_paintbrush_handles/ 

+8><bFE;@K@FE8C$E;<G<E;<E:<8E;)8@M<8P<Jc +8><b#@E;J@>?K<M8CL<J.:@<E:<c

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 296/309

JG<<:? <?8M@FI@JD GIF989CP ;<J<IM<J @KJ FNE GFJK 8K JFD< GF@EK8J@KN8J8G<IM<IJ@FEF=I8K@FE8C@JD9LKK?@J@JEFKK?8KGFJK

`4FL :8CC @K bJ@CCPca PFL @EHL@I< `9LK ?FN ;F PFL know K?8K PFLI

9I8@E I<GI<J<EKJ M@JL8C @D8><J $J @K D<I<CP K?8K PFL :8E :CFJ< PFLI<P<J8E;J<<K?<Da

/?@J HL<JK@FE used KF 9< ?8I;<I KF 8EJN<I 98:B @E K?< ;8P F= K?<:FEKIFM<IJP $= PFL N8EK<; KF GIFM< K?< <O@JK<E:< F= D<EK8C @De8><IP `J:@<EK@=@:8CCPa I8K?<I K?8E ALJK 9P @EKIFJG<:K@FE PFL ?8; KF@E=<I K?< <O@JK<E:< F= D<EK8C @D8><IP =IFD <OG<I@D<EKJ C@B< <>.?FN JL9A<:KJ KNF F9A<:KJ 8E; 8JB K?<D @= FE< :8E 9< IFK8K<; @EKF:FII<JGFE;<E:< N@K? K?< FK?<I /?< I<JGFEJ< K@D< @J C@E<8ICP GIFeGFIK@FE8C KF K?< 8E>C< F= IFK8K@FE I<HL@I<; /?@J @J <8JP KF <OGC8@E

@= PFL 8I< 8:KL8CCP M@JL8C@Q@E> K?< @D8>< 8E; :FEK@ELFLJCP IFK8K@E> @K 8K 8 :FEJK8EK JG<<; 9LK ?8I; KF <OGC8@E @= PFL 8I< ALJK :?<:B@E> GIFGFJ@K@FE8C=<8KLI<JF=K?<@D8><

/F;8P N< :8E 8:KL8CCP E<LIF@D8>< K?< C@KKC< G@:KLI<J @E K?< M@JLe8C :FIK<O .F P<J PFLI 9I8@E I<8CCP ;F<J I<GI<J<EK 8 ;<K8@C<; @D8><F= N?8K @K J<<J FI @D8>@E<J .<< .K<G?<E &FJJCPEcJ Image and Brain:The Resolution of the Imagery Debate.

+8IK F= K?< I<8JFE G<FGC< ><K @E KIFL9C< N@K? NFI;J @J K?8K K?<P 

;FEFKI<8C@Q<?FNDL:?:FDGC<O@KPCLIBJ9<?@E;NFI;J8E PFL M@JL8C@Q< 8 `>I<<E ;F>a 8E PFL M@JL8C@Q< 8 `:?<<J< 8Ge

GC<a

`GGC<a @JEcK ALJK 8 J<HL<E:< F= KNF JPCC89C<J FI =@M< C<KK<IJ/?8KcJ8J?8;FN /?8KcJK?<K@GF=K?<K@><IcJK8@C

2FI;J FI I8K?<I K?< :FE:<GKJ 9<?@E; K?<D 8I< G8@EK9ILJ?e<JgPFL :8E LJ< K?<D KF ;I8N @D8><J @E PFLI FNE D@E; '@K<I8CCP ;I8N @= PFL <DGCFP :FE:<GKJ KF D8B< 8 G@:KLI< @E PFLI M@JL8C :FIeK<O E; 9P K?< LJ< F= J?8I<; C89<CJ PFL :8E I<8:? @EKF JFD<FE<<CJ<cJ D@E; 8E; >I8JG K?<@I G8@EK9ILJ?<J KF ;I8N G@:KLI<J @E their D@E;JgJB<K:?8C@KKC<>I<<E;F>@EK?<@IM@JL8C:FIK<O

LK ;FEcK K?@EB K?8K 9<:8LJ< PFL J<E; JPCC89C<J K?IFL>? K?< 8@IFI C<KK<IJ K?IFL>? K?< $EK<IE<K @K @J K?< JPCC89C<J FI K?< C<KK<IJ K?8K;I8N G@:KLI<J @E K?< M@JL8C :FIK<O /?8K K8B<J JFD< :FDGC<O @EeJKIL:K@FEJ K?8K NFLC;EcK =@K @E K?< J<HL<E:< F= C<KK<IJ `GGC<a @J 9PK<J 8E; ;I8N@E> 8 G@:KLI< F= 8E 8GGC< =IFD J:I8K:? NFLC; K8B<DFI<;8K8K?8EK?8K

-%&6+-!,'/'*+

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 297/309

`GGC<a @J D<I<CP K?< K8> 8KK8:?<; KF K?< KIL< 8E; NFI;C<JJ apple:FE:<GK N?@:? :8E G8@EK 8 G@:KLI< @E PFLI M@JL8C :FIK<O FI :FCC@;<

 N@K? `:?<<J<a FI I<:F>E@Q< 8E 8GGC< N?<E PFL J<< FE< FI K8JK< @KJ

8I:?<KPG< @E 8GGC< G@< D8P9< <M<E J<E; FLK K?< DFKFI 9<?8M@FI =FI<8K@E>8E8GGC<_

 E; @KcJ EFK 8J J@DGC< 8J ALJK :8CC@E> LG 8 G@:KLI< =IFD D<DFIP*I ?FN NFLC; PFL 9< 89C< KF M@JL8C@Q< :FD9@E8K@FEJ C@B< 8 `KI@8Ee

 >LC8I C@>?K9LC9ag@DGFJ@E> KI@8E>C<E<JJ FE C@>?K9LC9J B<<G@E> K?<<JJ<E:<F=9FK?<M<E@=PFLcM<E<M<IJ<<EJL:?8K?@E>@EPFLIC@=<

FEcK D8B< K?< D@JK8B< K?< 9<?8M@FI@JKJ D8;< /?<I<cJ =8IDFI< KF JG<<:? K?8E JFLE; @E 8@I /?< C89<CJ 8I< ALJK GF@EKe<IJgaCFFB @E D<DFIP 8I<8 a .FFE<I FI C8K<I N?<E PFLcI<

?8E;<; 8 GF@EK<I @K :FD<J K@D< KF ;<I<=<I<E:< @K 8E; 8:KL8CCP CFFB@ED<DFIP8I<8

2?8K;F<J8NFI;GF@EKKF

/'*++%&,$(!&,*-+&$+

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 298/309

27. Variable Question Fallacies

Followup to 2FI;J8J(<EK8C+8@EK9ILJ?#8E;C<J

 C9<IK `M<IPK@D<$cM<C@JK<E<;KF8KI<<=8CC@KD8;<8 JFLE;JF$cCC>L<JJK?8KFK?<IKI<<J=8CC@E>8CJFD8B<JFLE;J $;FEcK9<C@<M<K?<NFIC;:?8E><J8IFLE;N?<E$cDEFKCFFB@E>a8IIP `28@K8D@ELK< $=EFFE<?<8IJ@K?FN:8E@K9<8JFLE;a

2?@C< NI@K@E> K?< ;@8CF>L< F= C9<IK 8E; 8IIP @E K?<@I ;@JGLK<

FM<I N?<K?<I 8 =8CC@E> KI<< @E 8 ;<J<IK<; =FI<JK D8B<J 8 JFLE;$ JFD<K@D<J =FLE; DPJ<C= CFJ@E> <DG8K?P N@K? DP :?8I8:K<IJ $

 NFLC; JK8IK KF CFJ< K?< >LK =<<C F= N?P anyone NFLC; <M<I 8I>L< C@B<K?8K<M<EK?FL>?$c;J<<E@K?8GG<ED8EPK@D<J

*E K?<J< F::8J@FEJ $ NFLC; I<G<8K KF DPJ<C= `@K?<I K?< =8CC@E> KI<< D8B<J 8 JFLE; FI @K ;F<J EFKa KF I<JKFI< DP 9FIIFN<; J<EJ<F=@E;@>E8K@FE

h+ FI S+i @J EFK 8CN8PJ 8 I<C@89C< ?<LI@JK@: @= PFL JL9JK@KLK< 8Ie

9@KI8IP E>C@J? J<EK<E:<J =FI + `/?@J J<EK<E:< @J =8CJ<a :8EEFK 9<:FEJ@JK<EKCP M@<N<; 8J KIL< FI =8CJ< E; K?<E K?<I<cJ K?< FC; :C8JeJ@:`#8M<PFLJKFGG<;9<8K@E>PFLIN@=<a

)FN @= PFL 8I< 8 D8K?<D8K@:@8E 8E; FE< N?F 9<C@<M<J @E :C8JeJ@:8C hI8K?<I K?8E @EKL@K@FE@JK@:i CF>@: K?<I< 8I< N8PJ KF :FEK@EL<@EJ@JK@E> K?8K h+ FI S+i @J 8 K?<FI<D =FI <O8DGC< J8P@E> K?8K `/?@JJ<EK<E:<@J=8CJ<a@JEFK8J<EK<E:<

LK JL:? I<JFCLK@FEJ 8I< JL9KC< N?@:? JL==@:<J KF ;<DFEJKI8K<8 E<<; =FI JL9KC<KP 4FL :8EEFK ALJK 9LCC 8?<8; FE <M<IP F::8J@FE

 N@K?`@K?<I@K;F<JFI@K;F<JEcKa

.F;F<JK?<=8CC@E>KI<<D8B<8JFLE;FIEFKFI_

.LI<CP 3 FI @K ;F<J EFK 2<CC D8P9< if  @KcJ really K?<J8D< 3 K?< J8D< 8E; K?< J8D< 8E; $= 3 <M8CL8K<J KF FEJFD< F::8J@FEJ 8E; FE 8EFK?<I PFLI @E;@>E8K@FE D8P 9< D@JeGC8:<;

http://lesswrong.com/lw/oc/variable_question_fallacies/ 

+8><b2FI;J8J(<EK8C+8@EK9ILJ?#8E;C<Jc +8><b@JGLK@E><=@E@K@FEJc

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 299/309

/F <M<E 9<>@E :C8@D@E> K?8K h+ FI S+i FL>?K KF 9< 8 E<:<JJ8IP KILK? K?< JPD9FC + DLJK JK8E; =FI exactly K?< J8D< K?@E> @E 9FK??8CM<J F= K?< ;@C<DD8 `@K?<I K?< =8CC D8B<J 8 JFLE; FI EFKag9LK

@= C9<IKJFLE; @J EFK K?< J8D< 8J 8IIPJFLE; K?<I< @J EFK?@E> G8I8;FO@:8C 89FLK K?< KI<< D8B@E> 8E C9<IKJFLE; 9LK EFK 8 8IeIPJFLE;

h/?< @;@FD @J JFD<K?@E> $ G@:B<; LG @E DP ;8PJ =FI8MF@;@E> E8D<JG8:< :FCC@J@FEJ $= PFLcM< >FK KNF ;@==<I<EK G8:B8><JK?8K ;<=@E< 8 :C8JJ .FLE; PFL :8E NI@K< +8:B8><.FLE; KF JG<:@=P 

 N?@:? .FLE; PFL D<8E /?< @;@FD @J EFK N@;<CP BEFNE $ K?@EB N?@:?@J8G@KP9<:8LJ<$F=K<EN@J?$:FLC;LJ<@K@ENI@K@E>i

/?< M8I@89@C@KP D8P 9< JL9KC< C9<IK 8E; 8IIP D8P :8I<=LCCP 

 M<I@=P K?8K @K @J K?< J8D< KI<< @E K?< J8D< =FI<JK 8E; K?< J8D<F::8J@FE F= =8CC@E> ALJK KF <EJLI< K?8K K?<P I<8CCP ;F ?8M< 8 JL9JK8EeK@M< ;@J8>I<<D<EK 89FLK <O8:KCP K?< J8D< <M<EK E; K?<E =FI><KKF :?<:B K?8K K?<P 8I< D8K:?@E> K?@J <M<EK 8>8@EJK <O8:KCP K?< J8D<:FE:<GK

/?@EB 89FLK K?< >IF:<IP JKFI< K?8K PFL M@J@K DFJK F=K<E $J @KFE K?< C<=K J@;< F= K?< JKI<<K FI K?< I@>?K LK F= :FLIJ< K?<I< @J EF`the C<=K J@;<a F= K?< JKI<<K FECP your C<=K J@;< 8J PFL KI8M<C 8CFE> @K

=IFD JFD< G8IK@:LC8I ;@I<:K@FE (8EP F= K?< NFI;J N< LJ< 8I< I<8CeCP=LE:K@FEJF=@DGC@:@KM8I@89C<JJLGGC@<;9P:FEK<OK

$KcJ 8:KL8CCP FE< ?<:B F= 8 G8@E I<HL@I@E> FE< ?<:B F= 8 CFK F=  NFIB KF ?8E;C< K?@J B@E; F= GIF9C<D @E 8E IK@=@:@8C $EK<CC@><E:<GIF>I8D @EK<E;<; KF G8IJ< C8E>L8><gK?< G?<EFD<EFE >F@E> 9P K?<E8D<F=`JG<8B<I;<@O@Ja

`(8IK@E KFC; F9 K?< 9L@C;@E> N8J FE ?@J C<=Ka LK `C<=Ka @J8 =LE:K@FEeNFI; K?8K <M8CL8K<J N@K? 8 JG<8B<Ie;<G<E;<EK M8I@89C<@EM@J@9CP >I899<; =IFD K?< JLIIFLE;@E> :FEK<OK 2?FJ< `C<=Ka @J

D<8EKF9cJFI(8IK@EcJ/?< M8I@89C<J @E 8 M8I@89C< HL<JK@FE =8CC8:P F=K<E 8I<EcK E<8KCP C8e

9<C<;g@KcJEFK8JJ@DGC<8J`.8P;FPFLK?@EB5<HL8CJa

$= 8 E8D<JG8:< :FCC@J@FE @EKIF;L:<J KNF ;@==<I<EK :FE:<GKJ K?8KCFFB C@B< `K?< J8D< :FE:<GKa 9<:8LJ< K?<P ?8M< K?< J8D< E8D<gFI8 D8G :FDGI<JJ@FE @EKIF;L:<J KNF ;@==<I<EK <M<EKJ K?8K CFFB C@B<K?< J8D< <M<EK 9<:8LJ< K?<P ;FEcK ?8M< J<G8I8K< D<EK8C =@C<JgFI

+8><b/89FF4FLI2FI;Jc +8><b!8CC8:@<JF=FDGI<JJ@FEc

.*!$)-+,!'&$$!+

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 300/309

K?< J8D< =LE:K@FE <M8CL8K<J @E ;@==<I<EK :FEK<OKJgK?<E I<8C@KP @KeJ<C= 9<:FD<J GIFK<8E :?8E><89C< K C<8JK K?8KcJ N?8K K?< 8C>FeI@K?D =<<CJ C@B< =IFD @EJ@;< 4FLI D@E;cJ <P< J<<J K?< D8G EFK K?<

K<II@KFIP;@I<:KCP$= PFL ?8M< 8 HL<JK@FE N@K? 8 ?@;;<E M8I@89C< K?8K <M8CL8K<J KF

;@==<I<EK <OGI<JJ@FEJ @E ;@==<I<EK :FEK<OKJ @K feels like I<8C@KP @KJ<C= @J LEJK89C<gN?8K PFLI D@E;cJ <P< J<<J J?@=KJ 8IFLE; ;<G<E;@E> FE

 N?<I<@KCFFBJ

/?@J F=K<E :FE=LJ<J LE;<I>I8;L8K<J h8E; GFJKDF;<IE@JK GIF=<JeJFIJi N?F ;@J:FM<I 8 J<EK<E:< N@K? DFI< K?8E FE< @EK<IGI<K8K@FEK?<PK?@EBK?<P?8M<;@J:FM<I<;8ELEJK89C<GFIK@FEF=I<8C@KP

`*? DP >FJ? b/?< .LE >F<J 8IFLE; K?< 8IK?c @J KIL< =FI #LE>8 #LEK<I>8K?<I<I 9LK =FI D8I8 JKIFEFD<I b/?< .LE >F<J 8IFLE;K?< 8IK?c @J =8CJ< /?<I< @J EF =@O<; KILK?a /?< ;<:FEJKIL:K@FE F= K?@JJFG?FDFI@:E@KN@KK<IP@JC<=K8J8E<O<I:@J<KFK?<I<8;<I

 E; P<K <M<E $ @E@K@8CCP =FLE; DPJ<C= NI@K@E> `$= 3 @J FE JFD<F::8J@FEJ 8E; FE 8EFK?<I K?< J<EK<E:< b 3c D8P ?8M< EF=@O<; KILK?eM8CL<a /?<I< @J EFK one J<EK<E:< N@K? 8 variable KILK?e

 M8CL< ` 3a has no KILK?eM8CL< $K @J EFK 8 proposition, EFK P<KEFK 8J D8K?<D8K@:@8EJ ;<=@E< GIFGFJ@K@FEeE<JJ 8EP DFI< K?8E `

a @J 8 GIFGFJ@K@FE FI `!I<; ALDG<; FM<I K?<a @J 8 >I8DD8K@:8C J<EeK<E:<

LK K?@J =8CC8:P K<E;J KF JE<8B @E <M<E N?<E PFL 8CC<><;CP BEFN 9<KK<I9<:8LJ<N<CCK?8KcJ?FNK?< 8C>FI@K?D=<<CJ=IFD@EJ@;<

+8><b#FNEC>FI@K?D!<<CJ!IFD$EJ@;<c +8><b#FNEC>FI@K?D!<<CJ!IFD$EJ@;<c

-%&6+-!,'/'*+

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 301/309

28. 37 Ways That Words Can Be Wrong

Followup to: %LJK 89FLK <M<IP GFJK @E !<9IL8IP 8E; JFD< @E

(8I:?.FD< I<8;<I @J 9FLE; KF ;<:C8I< K?8K 8 9<KK<I K@KC< =FI K?@J GFJK

 NFLC; 9< ` 28PJ /?8K 4FL 8E 0J< 2FI;J 0EN@J<CPa FI `28PJ /?8K .L9FGK@D8C 0J< *= 8K<>FI@<J 8E #8M< )<>8K@M< .@;<==<:KJ*E4FLIF>E@K@FEa

LK FE< F= K?< GI@D8IP C<JJFEJ F= K?@J >@>8EK@: C@JK @J K?8K J8P@E> `/?<I<cJ EF N8P DP :?F@:< F= 3 :8E 9< bNIFE>ca @J E<8ICP 8CN8PJ 8E<IIFI @E GI8:K@:< N?8K<M<I K?< K?<FIP 4FL :8E 8CN8PJ 9< NIFE>

M<E N?<E @KcJ K?<FI<K@:8CCP @DGFJJ@9C< KF 9< NIFE> PFL :8E JK@CC9< NIFE> /?<I< @J E<M<I 8 "<Ke*LKe*=e%8@Ce!I<< :8I; =FI 8EPK?@E>  PFL;F /?8KcJC@=<

<J@;<J $ :8E ;<=@E< K?< NFI; `NIFE>a KF D<8E 8EPK?@E> $ C@B<e@KcJEFKC@B<8NFI;:8E9< wrong.

+<IJFE8CCP $ K?@EB @K HL@K< ALJK@=@<; KF LJ< K?< NFI; `NIFE>a N?<E

A word fails to connect to reality in the first place. $J.F:I8K<J8 =I8DJK<I 4<JFIEF h/?<+8I89C<F=K?<8>><I i

Your argument, if it worked, could coerce reality to go a different way by choosing a different word definition. .F:I8K<J@J8 ?LD8E8E;?LD8EJ9P;<=@E@K@FE8I<DFIK8C .F@=PFL;<=@E<;?LD8EJKFEFK9<DFIK8CNFLC;.F:I8K<JC@M<=FI<M<I h/?<+8I89C<F=#<DCF:Bi

You try to establish any sort of empirical proposition as being true“by definition”. .F:I8K<J@J8?LD8E8E;?LD8EJ9P ;<=@E@K@FE8I<DFIK8C .F@J@K8 logical truth @=N<

<DG@I@:8CCPGI<;@:KK?8K.F:I8K<JJ?FLC;B<<CFM<I@=?<;I@EBJ?<DCF:B $KJ<<DJC@B<K?<I<8I<CF>@:8CCPGFJJ@9C<EFEeJ<C=e:FEKI8;@:KFIPNFIC;JN?<I<.F:I8K<J;F<JEcKB<<CFM<IeN?<I<?<cJ@DDLE<KF?<DCF:B9P8HL@IBF= 9@F:?<D@JKIPJ8P 'F>@:8CKILK?J8I<KIL<@E8CCGFJJ@9C<

 NFIC;J8E;JFE<M<IK<CCPFL which GFJJ@9C<NFIC;PFLC@M<

http://lesswrong.com/lw/od/37_ways_that_words_can_be_wrong/ 

+8><b/?<+8I89C<F=K?<8>><Ic +8><b/?<+8I89C<F=#<DCF:Bc

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 302/309

@Ee8E;8EPK?@E>PFL:8E<JK89C@J?`9P;<=@E@K@FEa@J8 CF>@:8CKILK? h/?<+8I89C<F=#<DCF:Bi

You unconsciously slap the conventional label on something,

without actually using the verbal definition you just gave. 4FLBEFNG<I=<:KCPN<CCK?8KF9@J`?LD8Ea<M<EK?FL>?FE PFLI;<=@E@K@FEPFL:8EE<M<I:8CCF9`?LD8EaN@K?FLK=@IJKF9J<IM@E>?@DKF9<DFIK8C h/?<+8I89C<F= #<DCF:Bi

The act of labeling something with a word, disguises achallengable inductive inference you are making. $=K?<C8JK<>>eJ?8G<;F9A<:KJ;I8NE?8M<9<<E9CL<8E;K?<C8JK:L9<J;I8NE?8M<9<<EI<;@K@J8D8KK<IF=@E;L:K@FEKF

J8PK?@JILC<N@CC?FC;@EK?<=LKLI< LK@=PFL:8CCK?<9CL<<>>J`9C<>>Ja8E;K?<I<;:L9<J`IL9<JaPFLD8P I<8:?@EKFK?<98II<C=<<C8E<>>J?8G<8E;K?@EB`*?8 9C<>>a h2FI;J8J#@;;<E$E=<I<E:<J i

You try to define a word using words, in turn defined with ever- more-abstract words, without being able to point to an example.`2?8K@JI<;a `-<;@J8:FCFIa `2?8KcJ8:FCFIa `$KcJ8 GIFG<IKPF=8K?@E>a `2?8KcJ8K?@E> 2?8KcJ8 GIFG<IKPa $KE<M<IF::LIJKFPFLKFGF@EKKF8JKFGJ@>E8E;8E8GGC< hOK<EJ@FEJ8E;$EK<EJ@FEJ i

The extension doesn’t match the intension. 2<8I<EcK:FEJ:@FLJCP8N8I<F=FLI@;<EK@=@:8K@FEF=8I<;C@>?K@EK?<JBP8J`(8IJaN?@:?N@CCGIF989CP?8GG<EI<>8I;C<JJF= 

 PFLI8KK<DGKKF;<=@E<`(8IJa8J`/?<"F;F=28IahOK<EJ@FEJ8E;$EK<EJ@FEJ i

Your verbal definition doesn’t capture more than a tiny fractionof the category’s shared characteristics, but you try to reason as if 

it does. 2?<EK?<G?@CFJFG?<IJF=+C8KFcJ:8;<DP :C8@D<;K?8KK?<9<JK;<=@E@K@FEF=8?LD8EN8J8 `=<8K?<IC<JJ9@G<;a@F><E<JK?<PE@:@JJ8@;KF?8M<<O?@9@K<;8GCL:B<;:?@:B<E8E;;<:C8I<;`#<I<@J+C8KFcJ(8Ea /?<+C8KFE@JKJGIFDGKCP:?8E><;K?<@I;<=@E@K@FE

+8><b/?<+8I89C<F=#<DCF:Bc +8><b/?<+8I89C<F=#<DCF:Bc +8><b2FI;J8J#@;;<E$E=<I<E:<Jc

+8><bOK<EJ@FEJ8E;$EK<EJ@FEJc +8><bOK<EJ@FEJ8E;$EK<EJ@FEJc

-%&6+-!,'/'*+

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 303/309

KF`8=<8K?<IC<JJ9@G<;N@K?9IF8;E8@CJa h.@D@C8I@KP CLJK<IJ i

You try to treat category membership as all-or-nothing, ignoring 

the existence of more and less typical subclusters. L:BJ8E;G<E>L@EJ8I<C<JJKPG@:8C9@I;JK?8EIF9@EJ8E;G@><FEJ$EK<I<JK@E>CP89<KN<<Ee>IFLGJ<OG<I@D<EKJ?FN<;K?8KJL9A<:KJK?FL>?K8;@J<8J<N8JDFI<C@B<CPKFJGI<8;=IFDIF9@EJKF;L:BJFE8E@JC8E;K?8E=IFD;L:BJKFIF9@EJh/PG@:8C@KP8E;JPDD<KI@:8C.@D@C8I@KP i

A verbal definition works well enough in practice to point out theintended cluster of similar things, but you nitpick exceptions. )FK<M<IP?LD8E?8JK<E=@E><IJFIN<8IJ:CFK?<JFILJ<J

C8E>L8><9LK@=PFLCFFB=FI8E<DG@I@:8C:CLJK<IF=K?@E>J N?@:?J?8I<K?<J<:?8I8:K<I@JK@:JPFLcCC><K<EFL>?@E=FID8K@FEK?8KK?<F::8J@FE8CE@E<e=@E><I<;?LD8E

 NFEcK=FFCPFL h/?<CLJK<I.KIL:KLI<F=/?@E>JG8:<i You ask whether something “is” or “is not” a category member but 

can’t name the question you really want answered. 2?8K@J8 `D8Ea $J8IE<PK?<89PFP8`D8Ea /?<`:FII<:Ka8EJN<ID8P;<G<E;:FEJ@;<I89CPFEN?<K?<IK?<HL<IP 

 PFL really N8EK8EJN<I<;@J`2FLC;?<DCF:B9<8>FF;K?@E>KF=<<;8IE<PaFI`2@CC8IE<PD8B<8>FF;?LJ98E;a h@J>L@J<;,L<I@<J i

You treat intuitively perceived hierarchical categories like theonly correct way to parse the world, without realizing that other  forms of statistical inference are possible even though your brain doesn’t use them. $KcJDL:?<8J@<I for a human KFEFK@:< N?<K?<I8EF9A<:K@J8`9C<>>aFI`IL9<aK?8E for a humanKFEFK@:<K?8KI<;F9A<:KJE<M<I>CFN@EK?<;8IB9LKI<;

=LII<;F9A<:KJ?8M<8CCK?<FK?<I:?8I8:K<I@JK@:JF=9C<>>J*K?<IJK8K@JK@:8C8C>FI@K?DJNFIB;@==<I<EKCP h)<LI8C8K<>FI@<J i

You talk about categories as if they are manna fallen from the Platonic Realm, rather than inferences implemented in a real brain. /?<8E:@<EKG?@CFJFG?<IJJ8@;`.F:I8K<J@J8D8Ea

+8><b.@D@C8I@KPCLJK<IJc +8><b/PG@:8C@KP8E;JPDD<KI@:8C.@D@C8I@KPc +8><b/?<CLJK<I.KIL:KLI<F=/?@E>JG8:<c

+8><b@J>L@J<;,L<I@<Jc +8><b)<LI8C8K<>FI@<Jc

/1+,,/'*+&/*'&

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 304/309

EFK`(P9I8@EG<I:<GKL8CCP:C8JJ@=@<J.F:I8K<J8J8D8K:?8>8@EJKK?<b?LD8Ec:FE:<GKa h#FNEC>FI@K?D!<<CJ!IFD$EJ@;< i

You argue about a category membership even after screening off  all questions that could possibly depend on a category-based inference. =K<IPFLF9J<IM<K?8K8EF9A<:K@J9CL<<>>eJ?8G<;=LII<;=C<O@9C<FG8HL<CLD@E<J:<EK8E;G8CC8;@LDe:FEK8@E@E>N?8KcJ left KF8JB9P8I>L@E>`$J@K8 9C<>>a LK@=PFLI9I8@EcJ:8K<>FI@Q@E>E<LI8CE<KNFIB:FEK8@EJ8hD<K8G?FI@:8Ci:<EKI8CLE@K:FII<JGFE;@E>KFK?<@E=<I<E:<F=9C<>>eE<JJ@KD8PJK@CC feel C@B<K?<I<cJ8 C<=KFM<IHL<JK@FE h#FNEC>FI@K?D!<<CJ!IFD

$EJ@;< i You allow an argument to slide into being about definitions, even

though it isn’t what you originally wanted to argue about. $=9<=FI<8;@JGLK<JK8IK<;89FLKN?<K?<I8KI<<=8CC@E>@E8 ;<J<IK<;=FI<JKD8B<J8`JFLE;aPFL8JB<;K?<KNFJFFEeKFe9<8I>L<IJN?<K?<IK?<PK?FL>?K8 JFLE;aJ?FLC;9<;<=@E<;8J`8:FLJK@:M@9I8K@FEJaFI`8L;@KFIP<OG<I@<E:<JaK?<Pc;GIF989CPK<CCPFLKF=C@G8:F@E *ECP8=K<IK?<8I>LD<EKJK8IKJ;F<JK?<;<=@E@K@FEF=8NFI;9<:FD<GFC@K@:8CCP:?8I><; h@JGLK@E><=@E@K@FEJ i

You think a word has a meaning, as a property of the word itself; rather than there being a label that your brain associates to a particular concept. 2?<EJFD<FE<J?FLKJ`4@B<J K@><Ia<MFCLK@FENFLC;EFK=8MFI8EFI>8E@JDK?8KK?@EBJ`#D_$?8M<ALJK?<8I;K?<JPCC89C<Jb/@<c8E;b"IIcN?@:?DP=<CCFNKI@9<D<D9<IJ8JJF:@8K<N@K?K?<@I@EK<IE8C8E8CF>L<JF=DPFNE tiger :FE:<GK8E;N?@:? aiiieeee

-0)#-0)#"0'+a .FK?<9I8@EK8B<J8 J?FIK:LK8E;@KJ<<DJK?8KK?<D<8E@E>F=K@><IE<JJ@J8 GIFG<IKPF=K?<C89<C@KJ<C= +<FGC<8I>L<89FLKK?< correct  meaning F=8C89<CC@B<`JFLE;ah!<<CK?<(<8E@E> i

You argue over the meanings of a word, even after all sidesunderstand perfectly well what the other sides are trying to say./?<?LD8E89@C@KPKF8JJF:@8K<C89<CJKF:FE:<GKJ@J8KFFC

+8><b#FNEC>FI@K?D!<<CJ!IFD$EJ@;<c +8><b#FNEC>FI@K?D!<<CJ!IFD$EJ@;<c

+8><b@JGLK@E><=@E@K@FEJc +8><b!<<CK?<(<8E@E>c

-%&6+-!,'/'*+

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 305/309

=FI:FDDLE@:8K@FE 2?<EG<FGC< want KF:FDDLE@:8K< N<cI<?8I;KFJKFG@=N<?8M<EF:FDDFEC8E>L8><N<cCC;I8NG@:KLI<J@EJ8E; 2?<EPFL<8:?LE;<IJK8E;N?8K@J

@EK?<FK?<IcJD@E;PFL8I< done. h/?<I>LD<EK!IFDFDDFE0J8>< i You pull out a dictionary in the middle of an empirical or moral 

 argument. @:K@FE8IP<;@KFIJ8I<?@JKFI@8EJF=LJ8><EFKC<>@JC8KFIJF=C8E>L8>< $=K?<:FDDFE;<=@E@K@FE:FEK8@EJ8GIF9C<De@=`(8IJa@J;<=@E<;8JK?<"F;F=28IFI8 `;FCG?@Ea@J;<=@E<;8J8B@E;F==@J?FI`)<>IF<Ja8I<;<=@E<;8J8J<G8I8K<:8K<>FIP=IFD?LD8EJK?<;@:K@FE8IPN@CCI<=C<:KK?<JK8E;8I;D@JK8B< h/?<

 I>LD<EK!IFDFDDFE0J8>< i You pull out a dictionary in the middle of any argument ever.

.<I@FLJCPN?8KK?<?<:BD8B<JPFLK?@EBK?8K;@:K@FE8IP <;@KFIJ8I<8E8LK?FI@KPFEN?<K?<I`8K?<@JDa@J8 `I<C@>@FEaFIN?8K<M<I $=PFL?8M<8EPJL9JK8EK@M<@JJL<

 N?8KJF<M<I8KJK8B<;FPFLI<8CCPK?@EB;@:K@FE8IP<;@KFIJ?8M<8::<JJKFLCK@D8K<N@J;FDK?8KJ<KKC<JK?<8I>LD<EKh/?<I>LD<EK!IFDFDDFE0J8>< i

You defy common usage without a reason, making it gratuitously hard for others to understand you. !8JKJK8E;LGGCLKFE@LD N@K?98><CJN@K?FLK?8E;C< h/?<I>LD<EK!IFDFDDFE0J8>< i

You use complex renamings to create the illusion of inference. $J8 `?LD8Ea;<=@E<;8J8`DFIK8C=<8K?<IC<JJ9@G<;a /?<E

 NI@K< `CCjDFIK8C=<8K?<IC<JJ9@G<;Jk8I<DFIK8C.F:I8K<J@J8jDFIK8C=<8K?<IC<JJ9@G<;kK?<I<=FI<.F:I8K<J@JDFIK8Ca 'FFBJC<JJ@DGI<JJ@M<K?8KN8P;F<JEcK@K

hDGKP'89<CJ i You get into arguments that you could avoid if you just didn’t use

the word. $=C9<IK8E;8IIP8I<EcK8CCFN<;KFLJ<K?< NFI;`JFLE;aK?<EC9<IKN@CC?8M<KFJ8P`KI<<=8CC@E> @E8;<J<IK<;=FI<JK><E<I8K<J8:FLJK@:M@9I8K@FEJa8E;8IIPN@CCJ8P`KI<<=8CC@E>@E8;<J<IK<;=FI<JK><E<I8K<J

+8><b/?<I>LD<EK=IFDFDDFE0J8><c +8><b/?<I>LD<EK=IFDFDDFE0J8><c +8><b/?<I>LD<EK=IFDFDDFE0J8><c

+8><b/?<I>LD<EK=IFDFDDFE0J8><c +8><bDGKP'89<CJc

/1+,,/'*+&/*'&

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 306/309

EF8L;@KFIP<OG<I@<E:<Ja 2?<E8NFI;GFJ<J8GIF9C<DK?<J@DGC<JKJFCLK@FE@JKF<C@D@E8K<K?<NFI;8E;@KJJPEFEPDJ h/89FF4FLI2FI;J i

The existence of a neat little word prevents you from seeing the details of the thing you’re trying to think about. 2?8K8:KL8CCP  >F<JFE@EJ:?FFCJFE:<PFLJKFG:8CC@E>@K`<;L:8K@FEa2?8KcJ8;<>I<<FE:<PFLJKFG:8CC@E>@K8 ;<>I<<a $=8 :F@EC8E;J`?<8;JaN?8KcJ@KJI8;@8CFI@<EK8K@FE 2?8K@J`KILK?a@=PFL:8EcKJ8P`8::LI8K<aFI :FII<:KaFI`I<GI<J<EKaFI`I<=C<:KaFI`J<D8EK@:aFI`9<C@<M<aFI`BEFNC<;><aFI`D8GaFI`I<8CaFI8EPFK?<IJ@DGC<K<IDh-<GC8:<K?<.PD9FCN@K?K?<.L9JK8E:< i

You have only one word, but there are two or more different things-in-reality, so that all the facts about them get dumped into a single undifferentiated mental bucket. $KcJG8IKF=8 ;<K<:K@M<cJFI;@E8IPNFIBKFF9J<IM<K?8K8IFCNFI<I<;C8JKE@>?KFIK?8KJ?<?8J9C8:B?8@I8E;@KcJG8IKF=8 ;<K<:K@M<cJFI;@E8IPNFIBKFNFE;<I@=D8P9<8IFC;P<J?<I?8@I LK@KK8B<J8JL9KC<I;<K<:K@M<KFNFE;<I@= K?<I<8I<KNF8IFCJJFK?8KK?<8IFCN?FNFI<I<;@JEFKK?<J8D<8JK?<8IFCN?F?8;9C8:B?8@I h!8CC8:@<JF=FDGI<JJ@FE i

You see patterns where none exist, harvesting other characteristics from your definitions even when there is no similarity along that dimension. $E%8G8E@K@JK?FL>?KK?8KG<FGC<F=9CFF;KPG<8I<<8IE<JK8E;:I<8K@M<9CFF;KPG<J8I<N@C;8E;:?<<I=LC9CFF;KPG<*J8I<8>I<<89C<8E;JF:@89C<8E;9CFF;KPG<J8I<:FFC8E;:FEKIFCC<;h8K<>FI@Q@E>#8JFEJ<HL<E:<J i

You try to sneak in the connotations of a word, by arguing from a definition that doesn’t include the connotations. `N@>>@Ea@J;<=@E<;@EK?<;@:K@FE8IP8J8G<IJFEN@K?>I<<E<P<J8E;9C8:B?8@I /?<NFI;`N@>>@Ea8CJF:8II@<JK?<:FEEFK8K@FEF=JFD<FE<N?F:FDD@KJ:I@D<J8E;C8LE:?<J:LK<989PJHL@II<CJ9LKK?8KG8IK@JEcK@EK?<;@:K@FE8IP .FPFLGF@EKKFJFD<FE<8E;J8P `"I<<E

+8><b/89FF4FLI2FI;Jc +8><b-<GC8:<K?<.PD9FCN@K?K?<.L9JK8E:<c

+8><b!8CC8:@<JF=FDGI<JJ@FEc +8><b8K<>FI@Q@E>#8JFEJ<HL<E:<Jc

-%&6+-!,'/'*+

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 307/309

<P<J C8:B?8@I .<<KFC;PFL?<cJ8N@>>@E 28K:?E<OK?<cJ>F@E>KFJK<8CK?<J@CM<IN8I<a h.E<8B@E>@EFEEFK8K@FEJ i

You claim “X, by definition, is a Y!” On such occasions you’re almost certainly trying to sneak in a connotation of Y that wasn’t in your given definition. 4FL;<=@E<`?LD8Ea8J8 `=<8K?<IC<JJ9@G<;a8E;[email protected]:I8K<J8E;J8P`)F=<8K?<IJeKNFC<>Je?<DLJK9<?LD8Ea LKN?8KPFL really :8I<89FLK@JJFD<K?@E><CJ<C@B<DFIK8C@KP $=N?8K N8J@E;@JGLK<N8J.F:I8K<JcJELD9<IF=C<>JK?<FK?<I=<CCFNNFLC;ALJKI<GCP`2?8;;8P8D<8E.F:I8K<JcJ>FKKNFC<>J /?8KcJN?8KN<cI<8I>L@E>89FLK@EK?<=@IJK

GC8:<a hI>L@E>`P<=@E@K@FEa i You claim “Ps, by definition, are Qs!”  $=PFLJ<<.F:I8K<JFLK

@EK?<=@<C;N@K?JFD<9@FCF>@JKJ>8K?<I@E>?<I9JK?8KD@>?K:FE=<II<J@JK8E:<KF?<DCF:BK?<I<cJEFGF@EK@E8I>L@E>`(<E9P;<=@E@K@FE8I<DFIK8Ca /?<D8@EK@D<

 PFL=<<CK?<E<<;KFK@>?K<EK?<M@J<9P@EJ@JK@E>K?8KJFD<K?@E>@JKIL<`9P;<=@E@K@FEa@JN?<EK?<I<cJFK?<I@E=FID8K@FEK?8K:8CCJK?<;<=8LCK@E=<I<E:<@EKF;FL9KhI>L@E>`P<=@E@K@FEa i

You try to establish membership in an empirical cluster “by definition”. 4FLNFLC;EcK=<<CK?<E<<;KFJ8P`#@E;L@JDby definition, @J8I<C@>@FEa9<:8LJ<N<CCF=:FLIJ<#@E;L@JD@J8I<C@>@FE $KcJEFKALJK8I<C@>@FE`9P ;<=@E@K@FEa@KcJC@B<8E actual I<C@>@FE K?<@JD;F<JEFKI<J<D9C<K?<:<EKI8CD<D9<IJF=K?<`I<C@>@FEa:CLJK<IJF@=@KN8JEcK=FIK?<=8:KK?8K8K?<@JD@J8I<C@>@FE by definition, PFLD@>?K>F8IFLE;K?@EB@E>K?8K8K?<@JD

wasn’t 8I<C@>@FE /?8KcJN?PPFLcM<>FKKF:ILJ?8CCFGGFJ@K@FE9PGF@EK@E>FLKK?8K`K?<@JD@J8I<C@>@FEa@JKIL< by definition9<:8LJ<@K@JEcKKIL<8EPFK?<IN8PhI>L@E>`P<=@E@K@FEa i

Your definition draws a boundary around things that don’t reallybelong together. 4FL:8E:C8@D@=PFLC@B<K?8KPFL8I< defining K?<NFI;`=@J?aKFI<=<IKFJ8CDFE>LGG@<JJ?8IBJ

+8><b.E<8B@E>@EFEEFK8K@FEJc +8><bI>L@E>`P<=@E@K@FEac

+8><bI>L@E>`P<=@E@K@FEac +8><bI>L@E>`P<=@E@K@FEac

/1+,,/'*+&/*'&

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 308/309

;FCG?@EJ8E;KIFLK9LKEFKA<CCP=@J?FI8C>8< 4FL:8E:C8@D@=PFLC@B<K?8KK?@J@JD<I<CP8C@JK8E;K?<I<@JEF

 N8P8C@JK:8E9<`NIFE>a *IPFL:8EJKFGGC8P@E>E@KN@K

 >8D<J8E;8;D@KK?8KPFLD8;<8D@JK8B<8E;K?8K;FCG?@EJ;FEcK9<CFE>FEK?<=@J?C@JK h2?<I<KFI8N K?<FLE;8IP i

You use a short word for something that you won’t need to describe often, or a long word for something you’ll need to describeoften. This can result in inefficient thinking, or even misapplications of Occam’s Razor, if your mind thinks that short  sentences sound “simpler”. 2?@:?JFLE;JDFI<GC8LJ@9C<`"F;;@;8D@I8:C<aFI`JLG<IE8KLI8CLE@M<IJ<e:I<8K@E> 

<EK@KPK<DGFI8I@CPJLJG<E;<;K?<C8NJF=G?PJ@:JahEKIFGP8E;.?FIKF;<J i

You draw your boundary around a volume of space where thereis no greater-than-usual density, meaning that the associated word does not correspond to any performable Bayesian inferences..@E:<>I<<Ee<P<;G<FGC<8I<EFKDFI<C@B<CPKF?8M<9C8:B?8@IFIM@:<M<IJ88E;K?<P;FEcKJ?8I<8EPFK?<I:?8I8:K<I@JK@:J@E:FDDFEN?P?8M<8NFI;=FI`N@>>@Eah(LKL8C$E=FID8K@FE8E;<EJ@KP@E/?@E>JG8:< i

You draw an unsimple boundary without any reason to do so./?<8:KF=;<=@E@E>8NFI;KFI<=<IKF8CC?LD8EJ<O:<GK9C8:BG<FGC<J<<DJB@E;F=JLJG@:@FLJ $=PFL;FEcKGI<J<EKI<8JFEJKF;I8NK?8KG8IK@:LC8I9FLE;8IPKIP@E>KF:I<8K<8E`8I9@KI8IPaNFI;@EK?8KCF:8K@FE@JC@B<8 ;<K<:K@M<J8P@E> `2<CC$?8M<EcKK?<JC@>?K<JKJ?I<;F= JLGGFIKFE<N8PFIK?<FK?<I=FIN?F:FLC;cM<DLI;<I<;K?FJ<FIG?8EJ_9LK?8M<N<:FEJ@;<I<;%F?E,

2@==C<?<@D8J8JLJG<:Ka h.LG<I<OGFE<EK@8CFE:<GKJG8:<8E;.@DGC<2FI;J i You use categorization to make inferences about properties that 

 don’t have the appropriate empirical structure, namely,conditional independence given knowledge of the class, to be well- approximated by Naive Bayes. )FN8P8D$KIP@E>KF

+8><b2?<I<KFI8NK?<FLE;8IPc +8><bEKIFGP8E;.?FIKF;<Jc

+8><b(LKL8C$E=FID8K@FE8E;<EJ@KP@E/?@E>JG8:<c +8><b.LG<I<OGFE<EK@8CFE:<GKJG8:<8E;.@DGC<2FI;Jc

-%&6+-!,'/'*+

7/27/2019 Lesswrong Sequences volume 1.0

http://slidepdf.com/reader/full/lesswrong-sequences-volume-10 309/309