The cities in Nigeria are safe as any city in a developing nation is. It's just if you are traveling overland things start getting dicey in Northern Nigeria. It doesn't get really bad until you get up a couple dozen miles north of Abuja, but I still wouldn't travel by land up there. Also, that little red line goes through Kogi which is a fairly violent area (even if none of that is aimed at tourists).
You should be fine in terms of "terrorism" with the places you are going to, but it is pretty easy to get caught in the middle of some violence with the Berber tribes (especially where there are large Taureg populations). The safest places are the cities where there is larger police and military presence, but if you are treking out in the middle of nowhere sometimes villages will get wiped out overnight due to tribal conflict, and that is not something you want to be around.
I'm with you on not believing the hype about violent areas when it comes to travel, but South Saharan Africa can get bad really fast, and you are getting painfully close to real problem areas in Mali and Nigeria, although not necessarily going into them.