In examining this issue of "what should I do", one commonly encounters people talking about counterfactual motivation.
There is, on this account, a distinction between what would fulfill an agent's actual desires, and what would fulfill the desires that an agent would have if she were fully informed, or fully rational, or meets some other counterfactual criterion.
I deny that these are different. Information, rationality, and the other criteria that these people mention have no influence on an agent's actual desires-as-ends.
Actual vs Counterfactual Motivation
Let's look at this claim more closely, then I will get back to issues that I see with it.
According to the Stanford Encyclopedia of Philosophy article, "Reasons for Action: Internal vs. External"
Another very important distinction among versions of Internalism is between Actual and Counterfactual versions. The former claim that if someone has a reason to do A, then it follows by necessity that she actually is somewhat motivated to do A (on the Motivation version), or actually has a desire that would be served by doing A (on the State version). Counterfactual versions make weaker claims: that if someone has a reason to do A, then it follows by necessity that she would be motivated to some degree, or would desire to do A, in circumstances of a particular kind.
Different Counterfactual theories disagree over the nature of this “particular kind” of circumstances: prominent proposals include (i) that the agent be in possession of full information, or at least not have any relevant false beliefs (Smith 1994; Joyce 2001); (ii) that she have completed “cognitive psychotherapy” or that her attitudes have reached a state of reflective equilibrium (Brandt 1979); (iii) that she have a vivid awareness of all relevant contingencies (Darwall 1983); (iv) that she deliberates faultlessly from her existing motivations (Williams 1979); (v) that she be practically rational (Korsgaard 1986)—a suggestion to which we shall return; and (vi) that she be ideally virtuous—a ‘phronimos’ (McDowell 1995).
Of these, the sixth option is different from the rest - so I wish to set it aside. The first five all suggest making adjustments to an agent's beliefs - that the agent be fully informed or aware of all relevant contingencies and the like. The last on the list actually calls for the agent to have different desires - good desires (virtues) and the absence of bad desires (vices). In almost every case, this set of desires would be different from the agent's actual desires.
My objection here is to counterfactuals that target beliefs.
Means and Ends
Much of what is written in this article seems to suggest that mistakes with regard to "means" of obtaining various ends is somehow a threat to the idea that reasons are grounded on actual desires. Thus, mistaken means are used as evidence that we are required to use some sort of "counterfactual" theory of reasons.
However, this is a mistake.
The mistake springs from the fact that the term "desire" (and similar terms such as "want" and "prefer" are, in fact, not only used when we talk about the ends or goals of intentional action, but also the means to those ends. A person wants a job so that she can make money so that she can afford an apartment where she can keep the temperature at a comfortable level. In this chain of desires, only the latter is desired as an end in itself or as a goal. Other things are desired as a means to that end.
False or irrational beliefs can get in the way of an agent getting what she really wants (desires-as-ends). Our agent may take a job thinking that it will allow her to pay for an apartment, only to discover that the job does not pay enough. She may purchase a condominium that she expects that she will be able to keep at a comfortable temperature only to discover that there is no air conditioning and the building rules prohibit individuals from installing air conditioning (because of old wiring).
However, when a person makes a mistake with respect to means, then it is NOT the case that the agent "has a reason to do X". In these types of cases, the agent in fact has no reason to do X. She only (and falsely) believes she has such a reason.
Note that in the original quote, the article identifies "actual" internalism as a theory that states that the agent "actually has a desire that would be served by doing A." However, in the case of a mistaken means, the agent does not, in fact, have a desire that would be served by doing A. The agent merely (and falsely) believes that she has a desire that is served by doing A. The proposition that doing A will serve a desire of hers is false.
To take another example. A person is thirsty reaches for a glass that she believes contains clean water, though in fact it is poisoned. She does not "really want" to drink out of that glass - she only thinks she does. A proper awareness of the fact will cause her to realize this mistake and put the glass down. It is perfectly legitimate to tell the agent reaching for the glass of water, "You don't want to do that." Then, when she asks "Why not?" tell her that the glass contains poison. She will respond by saying that you are correct, she did not want to do that.
Reasoning About Ends
Desires-as-ends are immune to reason.
You cannot reason a person out of a drug addiction or out of a sexual orientation. There is no syllogism that can prove that an agent's preference for chocolate over vanilla is mistaken. Desires-as-ends, like height, weight, age, and blood pressure, simply exist. You can no more reason a person to a new desire-as-end than you can reason her to a new height.
Consequently, "full information" or "vivid awareness of all relevant contingencies" or "cognitive therapy" is not going to change actual desires into something else.
They will allow the agent to choose more efficient means to an end, but they will not alter the ends (or the actual reasons) for intentional action.
These corrected beliefs can also be used to demonstrate to a person that she has reason to take steps to change some desire-as-end. In the same way that you can show a person that she has reason to lose weight, or to reduce her salt intake (to lower her blood pressure), you can reason her into taking steps to seek treatment for a drug addiction or to reduce his sex drive by showing her that the addiction is thwarting other desires (if, indeed, this is the case). Information (reason alone) will be as impotent in bringing about a direct change to harmful desires as they are in bringing about a direct change to weight or blood pressure. It takes "something else" to actually change these properties. However, true beliefs may inform a person who is otherwise mistaken that she has sufficient reason to go through the effort.
If we are going to argue that what an agent has a reason to do assumes that she has these corrected desires, I am going to ask whether it also means she should have the ideal weight and blood pressure. In all three cases, we are talking about an agent going through the effort to make a physiological change. Why are we to assume that the corrected desires matter but the corrected weight and blood pressure do not?
Besides, if we are talking about correcting these desires, we are leaving the set of counterfactuals concerned with corrected beliefs, which is outside of the scope of this particular discussion.
The section discussing counterfactual internalism ends with this statement:
Because it is uncontroversial that an agent can have reasons to do things that she is not actually motivated to do (particularly if she is unaware of those reasons), we will assume that interesting Motivation versions of internalism take Counterfactual forms.
It is only uncontroversial that an agent can have reasons to do things that she is not actually motivated to do if we mistaken means to be relevant to having reasons. A person has a reason to purchase a winning lottery ticket based on her ACTUAL desires - the desires that she can fulfill with the money from the winning lottery ticket. There is no different set of desires that she would have with more and better information. There is only the improved ability to determine the means towards those ends - the ability to choose the winning lottery numbers when buying the ticket.
So, as we try to answer the question, "What should I do?" - of course we are looking for whatever will actually fulfill desires, not for what an agent falsely believes would fulfill those desires. That is the whole point of asking the question. However, being fully informed, subjected to cognitive therapy, and perfect awareness of the alternatives are only going to be relevant in selecting the means to the ends, not the ends themselves. A person's actual ends, and the ends that the person would have if fully informed, are going to be the same.