Just feeling like refactoring is not a good reason to do it. Make an educated decision: Is the code change worth the effort? Stay straight on track, don't go astray.
Note: Usually, you'd refactor after finishing your story. At times, it might be necessary to refactor up front. But try to keep refactoring out of your current task.
Things should only be optimized a) when they have settled, i.e. haven't changed for a longer time, and b) when there is a measurable gain from the optimization. Ask yourself: Why would you want to optimize this? Will it bring a 10x speedup, or rather 1.5x? Where is the real bottleneck? Find the point where optimizing is worth the effort.
Sometime a task just cannot be completed in reasonable time. Keep an eye on effort and time. Is the time spent still reasonable? If it's not, think of alternatives.
Don't troll others with silly questions. Put some effort in solving problems yourself. However, if you're stuck, do ask for help.