'Min-maxing' is where you try to game a system to get the best possible stats, usually by analyzing the math behind it and squeezing every spare 0.1 points you can out of various skills, abilities, tools/weapons/armor, and other stats. The term itself I think means investing the minimum possible amount of points/effort/resources into getting the maximum results.
Personally, I don't like it. I think it is beyond the purpose of an RPG, which is story-telling interspersed with actual die-rolling. A good GM will recognize good role-playing and give bonuses (or hindrances) as needed. I think the fun is in the role-playing.
However, I knew a LOT of guys that loved doing this. For them it was more about number-crunching and doing the math. If that's their bag, more power to them. I won't disparage it, but in my opinion it just misses the point.