Hi Peter, you raised a couple of very interesting points here.
I will try to address them separately.
>> I think that it *might* speed up development, but not necessarily will.
I agree. One reason for that might be that TDD is a way of working which doesn't fit every project/human.
In my experience it takes quite a long time and practice to fully grasp the idea of TDD. I started writing "tests after" 8 years ago, then 2 years later I wrote some tests first after being introduced to TDD, but still wouldn't call myself a TDD practitioner at that time. I started reading more books on OOP and TDD and practicing a lot and I think I get the idea now of driving/designing a system outside in (Some call it ATDD).
At this stage I would certainly say that TDD speeds up development.
>> I think that TDD is first and most importantly about quality.
I agree. Additionally it gives you more focus on what you are doing, defect reduction (quality again?) and faster feature time to market (cycle time).
I would even say: TDD + Scrum = effectiveness to the max
>> I want to have the security of being able to refactor when necessary,
Definetely a benefit of TDD.
>> I want to have my different types of tests for their specific purpose,
>> but I am not really thinking about development performance.
Isn't Improving Quality (which TDD gives you) the only way to speed up your development?
BTW: I think that is the idea behind the Software Craftmanship movement:
"Only Quality lets you go faster"
http://manifesto.softwarecraftsmanship.org/