There's a tool called "perf" in linux, performance analyzer. Running it lists the functions where your program spends most of its time.
Generally you get 1 function which takes 90% of the time. Eliminating it makes the program run 10 times faster.
Then there's two functions taking 50% of the rest. Optimizing those makes the program run again twice as fast.
After that there's nothing more you can do. The rest can't be optimized or it's not worth trying.
Every program that I worked on, shows this trend.