If not for the drought, May would be too early. As it is you will likely get the brunt of the mosquitos that typically fill the air in spring. Last year, it was snowing in April but melted off fairly quickly. The snow currently in the mountains got a bit of a recharge the past couple of days, but not nearly enough. I won't lie - I want it to come down thick, many feet of snow, because water rationing is the next step and produce prices will go UP all over.
However - you can start from either of the mentioned trailheads, or from Happy Isles, and have a fine lower elevation trip regardless. The quota period is May through October - the full trailheads list here:
http://www.nps.gov/yose/planyourvisit/wildpermits.htm will be of use if you are hoping to reserve a permit in advance. If the snow melts and Glacier Point Road opens early, you would be able to use the Ostrander trailhead - a three day would allow a layover day at Ostrander to explore the area. Climbing Horse Ridge and wandering to the other end is a neat thing to do. The lakes might have a little ice left in them.
Memorial Day weekend will be slammed. I would get a permit from Sierra National Forest - Oakhurst office and drive out the bumpy road to Quartz Mountain trailhead, or Norris trailhead, and head to some lakes out thataway where the tourists don't tend to stampede. Get maps for roads and for the trails.